Skip to content

fix(ckpt): honor persistent retain intervals #20964

fix(ckpt): honor persistent retain intervals

fix(ckpt): honor persistent retain intervals #20964

Triggered via push August 25, 2026 05:36
Status Failure
Total duration 1h 29m 37s
Artifacts 41

cicd-main.yml

on: push
pre-flight  /  pre-flight
18s
pre-flight / pre-flight
configure
7s
configure
Lint check
27s
Lint check
cicd-wait-in-queue
5s
cicd-wait-in-queue
cicd-compute-build-matrix
3s
cicd-compute-build-matrix
Matrix: cicd-container-build
Matrix: cicd-unit-tests-core
Launch_Unit_Tests_Diffusion
2m 22s
Launch_Unit_Tests_Diffusion
generate-test-matrix
6s
generate-test-matrix
generate-gb200-test-matrix
8s
generate-gb200-test-matrix
Launch_Import_Check
39s
Launch_Import_Check
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-flaky
Nemo_CICD_Test
16s
Nemo_CICD_Test
Coverage_Fake
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
svcnvidia-nemo-ci
approved Aug 25, 2026
test

Automatically approved by queue manager

Annotations

4 errors, 86 warnings, and 41 notices
L0_Launch_models_deepseek
Process completed with exit code 1.
Result
❌ L0_Launch_models_deepseek — FAILED (exit 1)
L0_Launch_models_deepseek
Process completed with exit code 1.
Nemo_CICD_Test
Process completed with exit code 1.
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25, docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25, docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Launch_Unit_Tests_Diffusion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Diffusion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Core_2
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Core_2
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Core_0
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Core_0
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Core_1
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Core_1
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_generate
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_generate
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_megatron_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_megatron_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_callbacks
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_callbacks
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_decentralized_pg
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_decentralized_pg
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_flux
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_flux
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_distributed_save
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_distributed_save
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_dspark
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_dspark
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_megatron_mimo
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_megatron_mimo
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_local_checkpointing
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_local_checkpointing
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_resiliency
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_resiliency
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_pretrain
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_pretrain
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_tensor_inspect
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_tensor_inspect
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_finetune
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_finetune
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_wan_ckpt_conversion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_wan_ckpt_conversion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_recipes_deepseek_fsdp — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_converter_generate — PASSED
Result
✅ L0_Launch_megatron_fsdp — PASSED
Result
✅ L0_Launch_converter_fsdp — PASSED
Result
✅ L0_Launch_models_gemma4_vl — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_training_callbacks — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_training_decentralized_pg — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_training_distributed_save — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_training_dspark — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_training_local_checkpointing — PASSED
Result
✅ L0_Launch_training_resiliency — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_training_pretrain — PASSED
Result
✅ L0_Launch_training_tensor_inspect — PASSED
Result
✅ L0_Launch_training_finetune — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED

Artifacts

Produced during runtime
Name Size Digest
coverage-e2e-32813522994-0671c421-64e1-4890-98b5-b653d10702be
857 KB
sha256:e6af38de5a80e9f44636ae8c1957d83aa7c58bacbca79e14fc226a7a562e43cb
coverage-e2e-32813522994-1edfd0e0-32e9-41a5-b259-a0ff22dc5fd3
379 KB
sha256:9830850a61594aab11a22e3d0291884a9d50d298fb2b236ef49bc0c1bcbf59e6
coverage-e2e-32813522994-2aa71a16-77be-4b47-b67b-049bf0482722
389 KB
sha256:fb89491b06844d79700d8915bde6d82465b812e74f644a8bf128bed05ae43978
coverage-e2e-32813522994-2ad567e6-ad9a-413d-b244-68d26a33d439
802 KB
sha256:c12ec26cf261eedbc9a98d39054f0621aea0d006e389caa5844272e037ea1663
coverage-e2e-32813522994-33775c15-a473-41a9-b6da-b414f76f7b16
783 KB
sha256:dec0397cd1d2efe47a9d33070055159a16e8e88ad50e805aed335ac2ff4a9cd8
coverage-e2e-32813522994-341d6d30-da8b-4d8e-b498-35314cc6c806
286 KB
sha256:e52599dd879650668fd58d690508daa13686317e5084c7bd849b31d5e4d836f3
coverage-e2e-32813522994-375fc3c9-7cd9-4557-ba0b-9c4dcb561665
782 KB
sha256:d04cedbcbec496d2469fb9c41ffdfa930825935e8b36b84347c0b8e114f4371b
coverage-e2e-32813522994-3b767b8e-38f8-4ad6-9270-b9be11d334b0
386 KB
sha256:83415a388e1b9c834416540cd1d0d8b0fa3dcb5eac117d549f057296a314ed69
coverage-e2e-32813522994-4076a81c-fc13-4611-9cb1-611386996a7f
379 KB
sha256:a7eef0d06ee56fed0625096f9cc0deea7f991b13a3b720ad6d16efdc53dc95bb
coverage-e2e-32813522994-521319d3-6cd7-4165-9cd6-1d4b832fa58e
803 KB
sha256:6f74d759312461ad06376c20240dc0bcb56de52889236676c03b2e05e2cc7c0d
coverage-e2e-32813522994-55dc0be9-f78a-453b-9e16-bbe6254d24d9
286 KB
sha256:9df741ec9123950c411084b1a13a168323f900d2384abf0e9624e79b94662cdb
coverage-e2e-32813522994-58112d38-0d34-47d0-86b0-db19652fa38a
328 KB
sha256:d00be94fdb1f6bc7411532d011ccef04b171426e964bc87ee1b8705ae9281894
coverage-e2e-32813522994-5922bfc8-4b55-42d8-be25-2bfdf2cd7043
359 KB
sha256:4cf6f30882714c07228d36c2c16030a0f0ca73360a6baf5de55a71613e064577
coverage-e2e-32813522994-595c2ad7-a5e9-4767-a6fc-379103f61eec
357 KB
sha256:eb6a84dbfb3d1f4adcb312134eea938fd35c0f5768067ea02312e5ac1ca7b6c6
coverage-e2e-32813522994-62739168-2f92-49cb-879d-ec645a61e1db
359 KB
sha256:44ac5a27ce3dc47c8970b7b8bd80557f5ac57e6ba9e7b1e3cd27b25a985982cd
coverage-e2e-32813522994-6edcb0d6-2f9b-4380-b54b-c8ec959924c3
865 KB
sha256:8338c538582e44a6ffac7e4f755300268f292749feff73d285e25a9cfd263f77
coverage-e2e-32813522994-73426f02-40fa-4cfa-8c68-ce95fbb0cf1a
386 KB
sha256:ec7d981bc0f419471cffb036c07ae974e1a32119e8f785d757de58c2778bca8c
coverage-e2e-32813522994-7d12219a-d455-4f43-8a81-76b4eb2db8e0
432 KB
sha256:fb5ba4d2d8ee75fdaa2ba768c2c8b98ecf0b97615485a5c1d5c203cd6deeecc0
coverage-e2e-32813522994-82684412-f56c-4cda-aa5e-e5e84b023397
800 KB
sha256:83d8ab6ba9acb1f2f814a728d6679f29a2d3f7cececd591254df38e43f644839
coverage-e2e-32813522994-84f87ab1-4863-4d45-8a39-61632ba7aad9
286 KB
sha256:4a1fc64148d8af027edb5f80f6b6772dcb8ca3a2118683a456b43ebb379ec652
coverage-e2e-32813522994-86f2cc10-050e-4204-9d7b-b2884da03f4d
328 KB
sha256:aff1a65bdd0b915af3defa6474c830a8966bc9e2f19c563f2327388190ba4c20
coverage-e2e-32813522994-90ff0eb7-80ce-43cb-902e-aa9a8de29e85
857 KB
sha256:39e674994284bd733fa552eb729701ae284938cc2babbedb601085241d8b9b51
coverage-e2e-32813522994-97c6c030-c629-40ab-9e1f-bea86b565d76
785 KB
sha256:f7ecae6f84ba2b6f1817edb669ea272b1ffa83b98f40a2b4d7d3dcbeb9218dc5
coverage-e2e-32813522994-9b8dc626-9369-4a80-a6e5-ad4acbc3c7b0
389 KB
sha256:a62dbb8d70dea2eb91e126cb086c89f5e75110253c44bdbec16c2cd748e6ee4d
coverage-e2e-32813522994-a1daeed6-b38e-4fbb-a81c-06a2e8705893
1.29 KB
sha256:51f802eade3b02bbbd7e304722092ca8a80d28ce96755943c57947278ea06622
coverage-e2e-32813522994-a699a4ea-d9d4-4b11-920b-ce840cb8977c
400 KB
sha256:58ce65c69d4f7b9b4725a77ccc1739da35ba34accea3daeb3491b31111f44588
coverage-e2e-32813522994-ac0d8635-027a-449a-bf11-3a9004ea6194
432 KB
sha256:3b0473b6ef6c6ccc3808db9284579ebc6528410ebfcac605d60bfe9dc86e339f
coverage-e2e-32813522994-b4f24963-8258-4cf8-958f-00b225bcf974
419 KB
sha256:826897797bae4c67efa5e8c37aa7f5dbde71ccc160a0b2071a55ac2170e18b78
coverage-e2e-32813522994-c64ba96a-d45d-4180-99fc-00aa9558580f
400 KB
sha256:0a67807303858fed597861dbdc1562fab45648ac1ad2b2e93f7888371b1b2836
coverage-e2e-32813522994-d2120f9a-0df2-4b94-b33e-9174185a3e26
854 KB
sha256:f2ea3bb1fb7b078e3e807132817a6af67a0c7a6e6278cbaa5f86cd660c789b1e
coverage-e2e-32813522994-d3e34880-c645-42ba-8033-01c8996a1fc1
322 KB
sha256:da9e4261fb35268e8b8ed0ddf6af709eafdc24842b326c79b975cd06e50da0b3
coverage-e2e-32813522994-d8ba88f7-544b-423b-9bb9-5910be2940fe
783 KB
sha256:d7e1806ed24715a04a6f11a069f7c2f74afc990233e85ae6d6714dffcc23dffc
coverage-e2e-32813522994-e7259fd2-7447-4652-b535-ddc719bc15c6
501 KB
sha256:844b70e49ab4f346fcb2f36fa28fe73df3c6b62b4ca69386fb9bd9f44f4ec03e
coverage-e2e-32813522994-efaa017a-f75a-4ded-8864-e654646a624e
313 KB
sha256:a82e9f1e1421559ccbf9d88fd98fd0fccdb9c09d4a72ad46b09c71ba16331308
coverage-e2e-32813522994-f79ebf06-8b1e-475a-8cb0-78cc7e46d985
416 KB
sha256:668cbf799124174a6c16218b85218fe36ad45a1319a5620f844f918fbdeb4211
coverage-e2e-32813522994-fc666dec-c8ed-4ab5-9232-79dc9a96089c
330 KB
sha256:c0f115c51577bc6a493f12a670f4aac25449d775524ebc3ee71bfe8286d1b5aa
coverage-e2e-32813522994-ffbde6aa-3c26-4ce6-81b0-7e8702063aaa
559 KB
sha256:9ef4d0701ba89e78d2fc9ea09bdd3f30e0dc89eed4b85e6feff48d82b30eafac
coverage-unit-test-32813522994-01ad9e72-585f-455b-b967-c734c3a89fc5
945 KB
sha256:60b8480c59549db758eca3938a3819f56870a8ea0b7afc7b4cb360380bd6002b
coverage-unit-test-32813522994-22b83462-b28b-4866-8870-28e99f3b5e20
939 KB
sha256:06c8b87b869606d30042d4363c38400f3a546bd91f178d170c1427ebd93eb5bc
coverage-unit-test-32813522994-5df82176-e70b-4694-96d5-b377c14ad947
386 KB
sha256:50c8482f229b2ad5dfbb086c4490192057cea82e131cb1ef991096d8ca5e71bb
coverage-unit-test-32813522994-726705c9-0c36-4a5e-bfdf-72487c56f1b3
945 KB
sha256:c96ce7ec4d542cd1f5342295599db9afab8c7d0ab16af60306ece660f237f6f5