Unit Scaling: Out-of-the-Box Low-Precision Training

2023/03/20 by Charlie Blake, Blake, Charlie, Douglas J. Orr +3 · 5 citations
Computer Science · Physics and Astronomy · #Computational Physics and Python Applications #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2303.11257

openalex publication_date 2023/03/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present unit scaling, a paradigm for designing deep learning models that simplifies the use of low-precision number formats. Training in FP16 or the recently proposed FP8 formats offers substantial efficiency gains, but can lack sufficient range for out-of-the-box training. Unit scaling addresses this by introducing a principled approach to model numerics: seeking unit variance of all weights, activations and gradients at initialisation. Unlike alternative methods, this approach neither requires multiple training runs to find a suitable scale nor has significant computational overhead. We demonstrate the efficacy of unit scaling across a range of models and optimisers. We further show that existing models can be adapted to be unit-scaled, training BERT-Large in FP16 and then FP8 with no degradation in accuracy.

Cited by

• MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling
• Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding
• FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
• Elucidating the Design Space of FP4 training
• Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities

Related

• μnit Scaling: Simple and Scalable FP8 LLM Training
• A Metric Driven Approach to Mixed Precision Training
• Revisiting BFloat16 Training
• Scaling Laws for Precision
• WRPN & Apprentice: Methods for Training and Inference using Low-Precision Numerics