Skip to content

fix(ckpt): handle missing newer MLM checkpoint args - #5715

Open
caohch-1 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
caohch-1:fix/2127-legacy-mlm-args
Open

fix(ckpt): handle missing newer MLM checkpoint args#5715
caohch-1 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
caohch-1:fix/2127-legacy-mlm-args

Conversation

@caohch-1

Copy link
Copy Markdown

What does this PR do ?

Make MLM checkpoint config loading tolerate older checkpoints whose saved argparse.Namespace lacks newer optional Megatron-LM arguments.

Changelog

  • Use legacy-safe defaults when selecting MLA/heterogeneous config classes and when normalizing a missing cp_comm_type.
  • Avoid redundant direct reads of same-name TransformerConfig fields, and only map renamed checkpoint arguments when those source arguments are present.
  • Add regression coverage for missing newer arguments, coupled uneven-pipeline arguments, and preservation of explicitly set modern arguments.

Validation

  • Targeted tests/unit_tests/training/mlm_compat/test_arguments.py tests pass.
  • ruff check passes for both changed files.
  • ruff format --check passes for both changed files.
  • git diff --check passes.

GitHub Actions CI

This is an external contribution, so GitHub Actions CI will require approval from an NVIDIA developer.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation? No documentation changes are needed for this compatibility fix.
  • Does the PR affect components that are optional to install? No.
    • Reviewer: Does the PR have correct import guards for all optional libraries?

Additional Information

Fixes #2127

Signed-off-by: caohch-1 <caohch2023@shanghaitech.edu.cn>
@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@caohch-1
caohch-1 marked this pull request as ready for review August 20, 2026 12:40
@yaoyu-33 yaoyu-33 added area:ckpt Checkpoint conversion, loading, export, and save paths bug Something isn't working needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer labels Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ckpt Checkpoint conversion, loading, export, and save paths bug Something isn't working community-request needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Megatron-LM compatibility layer crashes with AttributeError on valid checkpoints missing newer args

2 participants