Skip to content

chore: AUT-1599 bump nightly Megatron Core pins - #5710

Open
svcnemo-autobot wants to merge 39 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:chore/bump-mcore-main-260819
Open

chore: AUT-1599 bump nightly Megatron Core pins#5710
svcnemo-autobot wants to merge 39 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:chore/bump-mcore-main-260819

Conversation

@svcnemo-autobot

@svcnemo-autobot svcnemo-autobot commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Background

Validate frozen MCore main 731b791469004f8fdcb896e65d610d6b1b0ebb32 and dev 2f2f8ebaef90f4d62a18a43808398223b1fc5809. Only this PR may merge; #5709 is validation-only.

What changed

  • Updated both commit files, the main gitlink, version-bound uv metadata, and the independently generated main lock.
  • Carried shared compatibility commit dd6e1cc51de7ad3eed3b0f11d0444aa3f755cdce.
  • Guarded GDN2 test coverage for MCore revisions that do not export it and applied the repository Ruff import spacing.

Details

flowchart LR
  C[compatibility dd6e1cc5] --> M[main landing lane]
  C --> D[dev validation lane]
  D -. joint proof .-> M
Loading

Both lanes retain the frozen pin pair and differ from compatibility only by gitlink, required uv metadata, and generated lock.

Tested

  • Exact Ruff autofix from the failing dev pre-commit gate applied
  • git diff --check
  • Commit files and main gitlink verified against the frozen pair
  • Paired local streams and current-head authoritative CI pending

@svcnemo-autobot
chore(deps): bump Megatron Core refs
bcdc43f 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
chore(deps): bump Megatron Core main
a6b80d4 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
fix(docker): keep source metadata writable
6650bf9 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge shared MCore compatibility fixes
79fdfd5
@svcnemo-autobot
Merge upstream main
4e9931c
@svcnemo-autobot
Merge shared MCore compatibility fixes
92f0ca7
@svcnemo-autobot
svcnemo-autobot requested a review from a team as a code owner August 20, 2026 09:28
@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

/ok to test 92f0ca7

@yaoyu-33 yaoyu-33 added area:build Dependencies, packaging, images, and environment setup ci CI, automation, test queue, or workflow infrastructure work needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer labels Aug 20, 2026
@svcnemo-autobot svcnemo-autobot mentioned this pull request Aug 21, 2026
Open
@svcnemo-autobot
chore(deps): use auditable MCore dev head
8572bf0 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge compatibility head 8572bf0
944730d
@svcnemo-autobot
fix(deps): restore frozen MCore dev pin
87011fd 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge compatibility head 87011fd
6025cf6 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
fix(ci): handle MCore runtime compatibility
83a3784 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge commit '83a37846b1d5cf77aa97d4ec7c10cee1fa6ffe51' of /workspace…
eae6a17 
…/compat into chore/bump-mcore-main-260819

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@ko3n1g

ko3n1g commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

/ok to test eae6a17

@svcnemo-autobot
fix(mcore): support both nightly compatibility lanes
2a990a3 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge compatibility head 2a990a3
e396530
@svcnemo-autobot
Merge upstream main into nightly compatibility
b9a66a3 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge commit 'b9a66a38cba514ff37b7b39979aef76b77eedbb2' into chore/bu…
93fbe33 
…mp-mcore-main-260819

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
chore(deps): record MCore main provenance
c12e315 
Frozen MCore commit: 731b791469004f8fdcb896e65d610d6b1b0ebb32
MCore tree: a2f96cc2302b68433c8fd0d82be5ca92a4d99fdd
MCore tree manifest SHA-256: afb51fccf5ecbc38d60f2f0ccf1fb53338196bfed9e94bff91b5a7256ceb4c8f
MCore pyproject blob: 4b4ad8c1c86b6adbee5d22578b160d21b6904345
MCore lock blob: 8a0dfaa9a3af1173496f1945137891c441fdc764
Bridge pyproject blob: 082fcb0
Bridge lock blob: 9827655

Resolved direct Git sources in the MCore lock:
- DeepGEMM 714dd1a4a980f7937a74343d19a8eba4fe321480
- Emerging-Optimizers 1effa026ff096b7fa1063ca2fba19d98be6e6cdf
- fast-hadamard-transform f134af63deb2df17e1171a9ec1ea4a7d8604d5ca
- FlashMLA b7643bd54521f563b839b98289b5cd048c062ba2
- mamba 0048fbf2e7b2f214dcbe703ea3dec2b9647595e1
- NeMo Lens b85578fc2b736a1804705e537001b5f45e9c715d
- NeMo Run e3935393a290aed1822af52139b4b8ee270fed1f
- TransformerEngine 4329ff84bfbdaa778a33cba02a15fb0807c64689

Bridge resolves the same TE, Emerging-Optimizers, FlashMLA, and Lens commits;
its fast-hadamard-transform v1.1.0 resolves to
1cc807efbd6cc001df359822d60bf6052dd66859.

Exact setup/build manifests and recursive tree manifests were inspected. Build
hooks compile extensions, initialize immutable submodules, or fetch project
GitHub release wheels; no dependency input was changed because the approval
finding identified no concrete malicious source or exfiltration mechanism.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

/ok to test c12e315

@svcnemo-autobot
test(training): guard optional GDN2 coverage
84f077e 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

/ok to test 1131dc3

@nemo-autobot-origin
nemo-autobot-origin Bot force-pushed the chore/bump-mcore-main-260819 branch from 1131dc3 to 604ee8a Compare August 22, 2026 16:18
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

/ok to test 604ee8a

@svcnemo-autobot nv-nemo-autobot-conversational

svcnemo-autobot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor Author

Joint nightly validation is complete.

  • Shared compatibility: dd6e1cc51de7ad3eed3b0f11d0444aa3f755cdce
  • Frozen MCore pair: main 731b791469004f8fdcb896e65d610d6b1b0ebb32, dev 2f2f8ebaef90f4d62a18a43808398223b1fc5809
  • Main head: 604ee8aa325db0287353eea86f3be3915139c513
  • Dev validation head: 40114284838143ae97520194e25e865c9bdd3874 (test: AUT-1599 validate Megatron Core dev compatibility #5709)
  • Both unit streams passed, including three core shards and 535 diffusion tests per lane
  • All 66 active H100 functional launchers passed in each lane; the quantization launcher passed with the required prepared offline dataset overlay
  • Main CICD NeMo passed in run 32584545902
  • Dev-authoritative CICD NeMo passed in run 32585007294
  • Each lane differs from the compatibility commit only in its permitted gitlink, lane-specific pyproject.toml metadata, and generated uv.lock

This PR is the sole merge candidate and is ready for human review. The dev PR must close as superseded after this PR lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area:build Dependencies, packaging, images, and environment setup ci CI, automation, test queue, or workflow infrastructure work full-test-suite needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@svcnemo-autobot @svcnvidia-nemo-ci @ko3n1g @yaoyu-33