Skip to content

Commit bbd7d02

Browse files
yeyu-nvidiaclaude
andcommitted
Import the MTP block's final_layernorm (repeated-MTP path)
The repeated-MTP import branch loops over the inner MTP layers but never imports the MTP block's own final_layernorm, leaving it random-initialized. Add the import after the loop, keyed to the last inner layer index (layer_id - 1) so it round-trips with the export side, which writes mtp.layers.{last}.final_layernorm. Bug found by @jenchen13. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 89ef596 commit bbd7d02

1 file changed

Lines changed: 5 additions & 0 deletions

File tree

modelopt/torch/export/plugins/megatron_importer.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -860,6 +860,11 @@ def _import_state_dict(self):
860860
)
861861

862862
layer_id += 1
863+
864+
# Import the MTP block's own final_layernorm into the last inner layer
865+
# index; without this it is left random-initialized. Mirrors the export
866+
# side, which writes mtp.layers.{last}.final_layernorm.
867+
self.rules["mtp.final_layernorm"](mtp.final_layernorm, layer_id - 1)
863868
else: # non-repeated MTP
864869
# MTP is the last layer in DeepSeek V3/R1
865870
layer_id += 1

0 commit comments

Comments
 (0)