2023/09/29 by Sergio P. Perez, Yan Zhang, Perez, Sergio P. +17 · 2 voices · 2 citations
Computer Science · #B.2.4 #Computation and Language (cs.CL) #Computational Physics and Python Applications #Emerging Technologies (cs.ET) #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #I.2.7 #Machine Learning (cs.LG) #Parallel Computing and Optimization Techniques #Performance (cs.PF) #Topic Modeling #cs.AR #cs.CL #cs.ET #cs.LG #cs.PF
paper · pdf · doi:10.48550/arxiv.2309.17224
openalex publication_date 2023/09/29 · arxiv published 2023/09/29 · arxiv updated 2023/09/29 · openalex created_date 2023/10/03 · openalex updated_date 2026/07/28
FP8 formats are gaining popularity to boost the computational efficiency for training and inference of large deep learning models. Their main challenge is that a careful choice of scaling is needed to prevent degradation due to the reduced dynamic range compared to higher-precision formats. Although there exists ample literature about selecting such scalings for INT formats, this critical aspect has yet to be addressed for FP8. This paper presents a methodology to select the scalings for FP8 linear layers, based on dynamically updating per-tensor scales for the weights, gradients and activations. We apply this methodology to train and validate large language models of the type of GPT and Llama 2 using FP8, for model sizes ranging from 111M to 70B. To facilitate the understanding of the FP8 dynamics, our results are accompanied by plots of the per-tensor scale distribution for weights, activations and gradients during both training and inference.