Pretraining Large Language Models with NVFP4

2025/09/29 by NVIDIA, Felix Abecassis, Abecassis, Felix +189 · 4 voices · 21 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2509.25149

openalex publication_date 2025/09/29 · arxiv published 2025/09/29 · openalex created_date 2025/10/19 · arxiv updated 2026/03/04 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) today are powerful problem solvers across many domains, and they continue to get stronger as they scale in model size, training set size, and training set quality, as shown by extensive research and experimentation across the industry. Training a frontier model today requires on the order of tens to hundreds of yottaflops, which is a massive investment of time, compute, and energy. Improving pretraining efficiency is therefore essential to enable the next generation of even more capable LLMs. While 8-bit floating point (FP8) training is now widely adopted, transitioning to even narrower precision, such as 4-bit floating point (FP4), could unlock additional improvements in computational speed and resource utilization. However, quantization at this level poses challenges to training stability, convergence, and implementation, notably for large-scale models trained on long token horizons. In this study, we introduce a novel approach for stable and accurate training of large language models (LLMs) using the NVFP4 format. Our method integrates Random Hadamard transforms (RHT) to bound block-level outliers, employs a two-dimensional quantization scheme for consistent representations across both the forward and backward passes, utilizes stochastic rounding for unbiased gradient estimation, and incorporates selective high-precision layers. We validate our approach by training a 12-billion-parameter model on 10 trillion tokens -- the longest publicly documented training run in 4-bit precision to date. Our results show that the model trained with our NVFP4-based pretraining technique achieves training loss and downstream task accuracies comparable to an FP8 baseline. These findings highlight that NVFP4, when combined with our training approach, represents a major step forward in narrow-precision LLM training algorithms.

Citations

• NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
• Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
• Recipes for Pre-training LLMs with MXFP8
• Rethinking the Outlier Distribution in Large Language Models: An In-depth Study
• FP4 All the Way: Fully Quantized Training of LLMs
• Quartet: Native FP4 Training Can Be Optimal for Large Language Models
• Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
• Oscillation-Reduced MXFP4 Training for Vision Transformers
• Training LLMs with MXFP4
• Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
• Systematic Outliers in Large Language Models
• Optimizing Large Language Model Training Using FP4 Quantization
• Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining
• Pushing the Limits of Large Language Model Quantization via the Linearity Theorem
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats
• Root Mean Square Layer Normalization
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Cited by

• MoR: Mixture Of Representations For Mixed-Precision Training
• Stable FP4 Training via Transposition-Invariant Block Quantization
• NVIDIA Nemotron 3: Efficient and Open Intelligence
• BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
• CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding
• MACKO: Sparse Matrix-Vector Multiplication for Low Sparsity
• TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control
• ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
• Mixed-Precision Quantization for Language Models: Techniques and Prospects
• HiFloat4 Format for Language Model Pre-training on Ascend NPUs
• Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
• O-POPE: High-Frequency Pipelined Outer Product based GEMM acceleration with minimal buffering overhead
• Direction-Preserving Number Representations
• Studying quantization trade-offs for efficient inference deployment in machine translation
• Price of metric universality in vector quantization is at most 0.11 bit
• The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
• Composer 2 Technical Report
• GNMR: Runtime Stability Control for Low-Precision Large Language Model Training
• Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference
• HiFloat4 Format for Language Model Inference
• Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery

Discussions

• NVIDIA just trained a 12B-parameter model on 10T tokens fully in 4-bit precision (NVFP4). It’s 2–3× faster uses 50% less memory than FP8, and matches accuracy (MMLU-Pro 62.58%). arxiv.org/abs/2509.251 [bsky, 2 points, 0 comments]
• Pretraining Large Language Models with NVFP4 [hn, 2 points, 0 comments]
• I know :) To pair that, see Nvidia's latest arxiv.org/abs/2509.25149 The ready-to-use models were mentioned because there is really not a reason to scale up and the architecture of large systems shoul [bsky, 1 points, 0 comments]
• Pretraining Large Language Models with NVFP4 [hn, 1 points, 0 comments]

Related

• Optimizing Large Language Model Training Using FP4 Quantization
• FP4 All the Way: Fully Quantized Training of LLMs
• Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
• TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control
• Stable FP4 Training via Transposition-Invariant Block Quantization