Low-Precision Training with Megatron-Bridge: FP8 (Current Scaling), MXFP8, NVFP4 vs BF16

[nemo-automation-bot[bot]]

Authors: @aditya_vavre, @nima_tajbakhsh

Introduction

In the ever-evolving field of deep learning, efficient training is crucial to handle larger models and datasets within practical resource constraints. Low-precision training has emerged as a powerful approach, enabling faster training and significant GPU memory savings without compromising model accuracy.

This post delves into recent advances in low-precision training, specifically focusing on FP8 per-tensor current scaling (FP8-CS), MXFP8, and NVFP4 precisions. We will compare these to the more established BF16 precision training and discuss the practical advantages of adopting these newer formats.

What is Low-Precision Training?

Low-precision training involves using numerical formats with fewer bits to represent weights, activations, and gradients during neural network training. This reduces memory bandwidth and computational demand, leading higher training throughput. NVIDIA Blackwell GPUs provide native support for general matrix multiplications (GEMMs) for a wide range of microscaling formats: MXFP8, MXFP6, MXFP4, NVFP4.

Overview of Precision Formats

Figure 1: Comparison of FP8, MXFP8 and NVFP4 precision formats.

FP8-CS

FP8-CS applies FP8 to linear layers via scaling factors based on the statistical properties of each tensor in the current training iteration. It is therefore more reactive to the immediate dynamic range of the tensor, potentially leading to more accurate quantization in each step. FP8-CS uses the E4M3 format for representing weights and activations, and the E5M2 format to represent gradients (due to its wide dynamic range).

MXFP8

MXFP8 is an exemplification of block scaling for the NVIDIA Blackwell architecture. With MXFP8, each contiguous block of 32 values in a tensor is assigned its own scaling factor, computed directly by the GPU’s Tensor Cores. All values are encoded in E4M3 datatype while the scaling factors themselves use the E8M0 datatype. This finer-grained scaling mechanism can better capture the data distribution during training and result in better model convergence.

NVFP4

NVFP4 further improves memory efficiency and throughput by using the E2M1 4-bit format. It maintains numerical accuracy across a wide dynamic range of tensor values by utilizing (1) a E4M3 scaling factor to each 16-value micro-block (2) a second-level FP32 scalar applied per tensor.

Empirical Comparison: Llama3-8B and Research-8B Models

To demonstrate the practical impact of low-precision training, we evaluate the convergence and downstream task performance of low-precision pretraining across two dense transformer model architectures, Llama3-8B and an NVIDIA internal research 8B model (Research-8B with dense grouped query attention (GQA) architecture that is similar to Llama 3 8B), trained on up to 1 trillion tokens. Our goal is to assess whether emerging low-precision formats can match the convergence behavior and final model quality of conventional BF16 training.

Experimental Setup

We ran a set of large-scale pretraining experiments using four different numeric precisions: BF16 (default baseline), FP8-CS, MXFP8 and NVFP4.

Note that MXFP8 and NVFP4 are natively supported ONLY on NVIDIA Blackwell GPUs. We trained Llama3-8B and Research-8B under identical settings (optimizer, learning rate schedule, etc.) so we could directly compare the impact of precision. Training was conducted using NeMo Megatron-Bridge on NVIDIA B200 GPUs with a batch size of 768, maximum learning rate of 6e-4, cosine decay to 6e-6, and a warmup of approximately 8.4B tokens.

For FP8-CS, we keep the first and last transformer layers in BF16 precision to maintain numerical stability. For NVFP4, following the setup and recipe from Pretraining Large Language Models with NVFP4, the final 4 transformer layers (<15% of total layers) are retained in BF16. We denote this as F0L4.

We experiment with two datasets: 1. Lingua DCLM Dataset which is a high-quality curated pretraining dataset sourced from Common Crawl. 2. Internal research dataset which is a blend of NVIDIA's released pretraining corpora. We trained Llama3-8B on both datasets and Research-8B on the internal research dataset.

Convergence Behavior

Let's start with training dynamics. The figures below show training and validation loss curves for all four precisions.

Figure 2: Training and validation loss curves for Llama3-8B model trained on the DCLM dataset. Training stability is maintained across precisions. Although NVFP4 shows slightly higher loss, downstream accuracies remain unaffected (see Table 1).

Figure 3: Training and validation loss curves for Llama3-8B model trained on the internal research dataset. Although NVFP4 shows slightly higher loss, downstream accuracies remain unaffected (see Table 1)

Figure 4: Training and validation loss curves for Research-8B model trained on the internal research dataset. Although NVFP4 shows slightly higher loss, downstream accuracies remain unaffected (see Table 1)

Across both models and datasets, the low-precision runs closely track the BF16 baseline in both training and validation loss. FP8-CS and MXFP8 almost perfectly overlap with BF16 throughout the entire training process, showing no signs of instability or divergence. NVFP4, despite operating at a much lower 4-bit precision, demonstrates stable convergence and achieves a final validation loss within ~1% of BF16. Despite the small gap in loss, downstream task accuracies remain largely unaffected as we show below.

Downstream Evaluation

We also measured how these models perform after pretraining on standard downstream benchmarks: MMLU (5-shot), Winogrande (5-shot), HellaSwag (10-shot; norm accuracy), and ARC-Challenge (25-shot; norm accuracy). All evaluations were run in BF16 precision to isolate the impact of training precision. We report the performance of the last checkpoint.

The results are shown in Table 1 and tell a consistent story: models trained with low precision achieve downstream accuracies nearly identical to the BF16 baseline. MXFP8 slightly outperforms FP8-CS, likely due to fine-grained scaling design offering better dynamic range within a tensor. NVFP4 maintains downstream accuracy within <1% of BF16 across all benchmarks.

Model	Dataset	Precision	MMLU (↑)	Hellaswag (↑)	Winogrande (↑)	ARC-C (↑)
Llama3-8B	DCLM	BF16	45.98	76.44	70.17	51.28
		FP8-CS	46	75.25	70.24	49.91
		MXFP8	46.56	75.46	71.27	51.11
		NVFP4	45.64	75.59	69.38	51.28
Llama3-8B	Research dataset	BF16	52.73	75.71	67.88	51.37
		FP8-CS	52.46	75.65	70.17	54.52
		MXFP8	53.7	75.54	69.69	51.62
		NVFP4	52.83	75.04	71.98	53.58
Research-8B	Research dataset	BF16	53	76.98	70.4	55.89
		FP8-CS	52.62	75.81	70.8	54.44
		MXFP8	52.38	76.55	69.77	53.58
		NVFP4	52.21	76.19	70.32	54.95

Table 1: Downstream task accuracy (in %) for Llama3-8B and Research-8B models. Despite small differences in training/validation loss, all precision formats achieve similar downstream task accuracy.

Task Performance vs. Tokens Consumed

To further analyze convergence in terms of knowledge acquisition, we plot MMLU accuracy against tokens consumed during pretraining in Figure 5. The curves show that all low-precision formats exhibit nearly the same learning trajectories to BF16 during pretraining. NVFP4 initially lags slightly during early training (<200B tokens) on the DCLM dataset, but catches up and converges to the same plateau by ~700B tokens. This demonstrates that NVFP4 preserves learning efficiency over long-horizon pretraining.

Figure 5: MMLU accuracy vs tokens consumed during pretraining.

Ablation Study: How Many Layers Need BF16 for Stable NVFP4 Training?

Although NVFP4 enables highly efficient 4-bit training, our experiments show that some layers must remain in BF16 precision to ensure stable optimization and strong downstream performance. To understand this behavior, we conducted an ablation over different FxLy configurations (where x is the number of initial layers kept in BF16, and y is the number of final layers kept in BF16).

We evaluated the following configurations: F0L0, F0L1, F0L2, F0L4, F1L1, and compared both their training dynamics (Figure 6) and MMLU performance (Figure 7).

Stability and Convergence

As shown in Figure 6, F0L0 (fully NVFP4) diverges early, indicating that NVFP4 requires keeping some layers in BF16 for stable training. All other configurations (F0L1, F0L2, F0L4, and F1L1) train stably, with smooth training and validation loss curves.

Effect of BF16 Layer Placement (F0L2 vs. F1L1)

To isolate whether BF16 matters more at the start or end of the network, we compare F0L2 (two BF16 layers at the end) with F1L1 (one at the beginning, one at the end). Both use two BF16 layers total. F1L1 shows higher training loss (1.84081 vs. 1.83577, +0.27%), higher validation loss (1.35356 vs. 1.35003, +0.26%), and slightly lower downstream MMLU accuracy throughout training (as shown in Figure 7). This consistent gap shows that BF16 layers near the end of the network have a greater impact on final model quality.

Effect of More BF16 Layers

Across all configurations, adding more BF16 layers improves convergence and downstream accuracy, but with diminishing returns. Keeping the last four layers in BF16 (F0L4) is sufficient to match the quality of heavier BF16 configurations while retaining the efficiency benefits of NVFP4.

Figure 6: Training and validation loss curves for Research-8B model trained on the internal research dataset for different FxLy configurations.

Figure 7: MMLU accuracy vs tokens consumed during pretraining for different FxLy configurations. FP8-CS shown as a baseline.

Ablation Study: Does Training Recipe Impact NVFP4 Convergence?

We ran a focused recipe ablation on Llama3-8B with the DCLM dataset using three pretraining configurations (see Table 2 below) to test how optimizer hyperparameters and batch/LR schedules interact with NVFP4 low-precision format.

Recipe Config (Learning Rate Max → Min | Global Batch Size | AdamW ε)	BF16	NVFP4 F0L4
Recipe0: LR 3e-4 → 3e-5 | GBS 512 | AdamW ε = 1e-5	44.06	27.48
Recipe1: LR 6e-4 → 6e-6 | GBS 768 | AdamW ε = 1e-8	47.30	46.10
Recipe2: LR 6e-4 → 6e-6 | GBS 768 | AdamW ε = 1e-5	45.71	43.63

Table 2: MMLU (5-shot) accuracy, reported as the maximum of the final three checkpoints for BF16 and NVFP4 across all recipe configurations.

Figure 8:Training loss across different recipe configurations for Llama3-8B on DCLM dataset.

Figure 8 shows that Recipe0 and Recipe2 converge significantly slower. The key issue is the AdamW $\epsilon$ term, which regularizes the denominator $\sqrt{v_t} + \epsilon$. When $\epsilon$ is large (1e-5), it dominates the denominator because $\sqrt{v_t}$ is very small due to gradient-norm clipping. This effectively removes second-moment adaptation:

• AdamW update: $w = w - \text{lr} \cdot \frac{m_t}{\sqrt{v_t} + \epsilon}$
• If $\epsilon \gg \sqrt{v_t}$, then $w \approx w - \frac{\text{lr}}{\epsilon} \cdot m_t$ (SGD with momentum)

Our experiments indicate that when using a $\epsilon$ value of 1e-5, the $\sqrt{v_t}$ term is approximately 1 order of magnitude smaller than $\epsilon$. Thus, a large $\epsilon$ collapses AdamW into SGD which means step magnitudes shrink (slow convergence) and loss of per-parameter normalization. SGD is much more sensitive to gradient noise because it no longer normalizes updates using the second moment; noisy gradients translate directly into noisy momentum, which in turn produces noisy weight updates (low signal to noise ratio). NVFP4 introduces additional quantization noise on top of the noise induced by large $\epsilon$. This amplified noise directly harms convergence and leads to degraded downstream accuracy. This is why Recipe0 performs poorly (small batch, large $\epsilon$, low LR → lowest SNR and the strongest SGD-like behavior). Recipe2, despite sharing the same $\epsilon$, performs better because its larger batch size reduces gradient noise, stabilizing updates and limiting NVFP4 drift, thus resulting in a smaller BF16–NVFP4 accuracy gap compared to Recipe0.

Key Insights

• Low precision formats (FP8, MXFP8, NVFP4) achieve pretraining and validation losses very close to BF16, showing minimal degradation.
• Downstream task accuracy remains virtually unaffected, demonstrating that low-precision training maintains model effectiveness.
• MXFP8 often performs slightly better than standard FP8, due to its finer-grained scaling mechanism.
• NVFP4, despite its aggressive compression, still delivers competitive results when properly calibrated. The empirical sweet spot in our runs is Recipe1 (AdamW $\epsilon$=1e-8, LR=6e-4 → 6e-6, GBS=768) paired with NVFP4-F0L4: it retains NVFP4’s efficiency while matching BF16 quality.

Advantages of Using FP8, MXFP8, NVFP4

Precision	Micro Batch Size	Throughput (TFLOPS/GPU)	Speedup vs BF16	Step Time (s)	Time to train 100B tokens (hr)	Time Reduction
BF16	2	1165	-	5.79	153	-
FP8-CS (F1L1)	2	1547	1.33	4.36	115	-25%
MXFP8	2	1540	1.32	4.38	116	-24%
NVFP4 (F0L4)	4	1850	1.59	3.64	96	-37%

Table 3: Performance comparison for Llama3-8B training (GBS=128, Seq. Length=8192). FxLy denotes first 'x' layers and last 'y' transformer block layers are kept in BF16 precision.

1. Faster End-to-End Training

Using 8-bit or 4-bit numeric formats drastically reduces computational overhead by allowing GPUs to process more operations per clock cycle. Table 3 shows that, on NVIDIA Blackwell (GB200) GPUs, lower-precision formats deliver substantial throughput improvements and shorter step times compared to traditional BF16 training. Specifically, FP8-CS achieves a 1.33× throughput improvement, and NVFP4 format pushes efficiency further, delivering 1.59× higher throughput than BF16.

2. GPU Memory Savings

Table 4 provides a detailed breakdown of memory usage across training components for different pretraining precisions. Using lower bit-width formats reduces the memory footprint of weights and activations, allowing larger models or batch sizes on the same hardware. As shown in Table 3, NVFP4's efficiency enables the micro-batch size to double (from 2 to 4) during pretraining, directly improving throughput and scalability.

			Optimizer
Precision	Parameter	Gradients	Momentum	Variance	Master Parameter	Others
FP16	FP16	FP32	FP32	FP32	FP32
BF16	BF16	BF16
FP8 (tensor scaling)	FP8x2	BF16				Scaling factor per weight tensor
MXFP8	FP8x2	BF16				(Scaling factor per 32 elements) x 2
NVFP4	FP4	BF16				16x16 2D block scales replicated for each 1x16 block

Table 4: Memory footprint across training components for different precision formats.

Challenges and Considerations

• Hardware Support: MXFP8 and NVFP4 formats require the adoption of NVIDIA Blackwell GPUs.
• BF16 Layer Requirements: Ablation studies show that fully NVFP4 models (F0L0) diverge, and stable training requires keeping some layers in BF16. BF16 layers near the end of the network are especially important for mitigating NVFP4 quantization error.
• Recipe Dependence: Stable NVFP4 performance depends on using well-tuned training recipes (e.g., small AdamW $\epsilon$, appropriate LR schedule, and sufficiently large batch size). Poorly tuned recipes exacerbate noise and lead to significant quality degradation.

Training with Low Precisions using Megatron-Bridge

Megatron-Bridge enables users to train their models in low-precision. Low-precision recipes for Llama3-8B are available here. You can use the recipes following the sample code below:

from megatron.bridge.recipes.llama import llama3_8b_low_precision_pretrain_config as low_precision_pretrain_config
from megatron.bridge.training.gpt_step import forward_step

precision = "bf16_with_fp8_current_scaling_mixed"  # should be one of ["bf16_with_mxfp8_mixed", "bf16_with_fp8_current_scaling_mixed", "bf16_with_nvfp4_mixed"]
cfg = low_precision_pretrain_config(
    mixed_precision_recipe = precision,
    train_iters = 100,
    lr_warmup_iters = 10,
    lr_decay_iters = 90,
    mock = True,  # use mock dataset
)
pretrain(config=cfg, forward_step_func=forward_step)

Try out our simple python notebook which showcases how one can use the low-precision recipes to speed up their training workloads: Quickstart Notebook

Conclusion

Low-precision training formats like FP8 (current scaling), MXFP8, and NVFP4 offer exciting new avenues for faster, more efficient deep learning training compared to the widely adopted BF16. Their advantages in speed and memory savings open doors for training larger, more complex models. Empirical evidence from Llama3-8B and Research-8B models confirms that training with low precisions can match BF16 performance on both pretraining metrics and downstream tasks. As software frameworks evolve, we expect these formats to become mainstream, pushing the boundaries of AI capabilities.