FP8-LM: Training FP8 Large Language Models

2023/10/27 by Houwen Peng, Kan Wu, Peng, Houwen +37 · 12 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.18313

openalex publication_date 2023/10/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we explore FP8 low-bit data formats for efficient training of large language models (LLMs). Our key insight is that most variables, such as gradients and optimizer states, in LLM training can employ low-precision data formats without compromising model accuracy and requiring no changes to hyper-parameters. Specifically, we propose a new FP8 automatic mixed-precision framework for training LLMs. This framework offers three levels of FP8 utilization to streamline mixed-precision and distributed parallel training for LLMs. It gradually incorporates 8-bit gradients, optimizer states, and distributed learning in an incremental manner. Experiment results show that, during the training of GPT-175B model on H100 GPU platform, our FP8 mixed-precision training framework not only achieved a remarkable 39% reduction in real memory usage but also ran 75% faster than the widely adopted BF16 framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer Engine by 37%. This largely reduces the training costs for large foundation models. Furthermore, our FP8 mixed-precision training methodology is generic. It can be seamlessly applied to other tasks such as LLM instruction tuning and reinforcement learning with human feedback, offering savings in fine-tuning expenses. Our FP8 low-precision training framework is open-sourced at https://github.com/Azure/MS-AMPaka.ms/MS.AMP.

Cited by

• BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
• Platinum: Path-Adaptable LUT-Based Accelerator Tailored for Low-Bit Weight Matrix Multiplication
• MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling
• FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error
• Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding
• TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control
• FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
• Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
• Compute-Optimal Quantization-Aware Training
• Metis: Training LLMs with FP4 Quantization
• Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?
• Foundation Models for Clean Energy Forecasting: A Comprehensive Review

Related

• Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs
• MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling
• Towards Fully FP8 GEMM LLM Training at Scale
• μnit Scaling: Simple and Scalable FP8 LLM Training
• Optimizing Large Language Model Training Using FP4 Quantization