2023/12/09 by Lin Wu, Felix Dangel, Lin, Wu +11 · 1 citation
Computer Science · Earth and Planetary Sciences · Physics and Astronomy · #FOS: Computer and information sciences #Geophysical and Geoelectrical Methods #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Matrix Theory and Algorithms #Model Reduction and Neural Networks
paper · pdf · doi:10.48550/arxiv.2312.05705
openalex publication_date 2023/12/09 · openalex created_date 2023/12/13 · openalex updated_date 2026/07/28
Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their preconditioning Kronecker factors are dense, and numerically unstable in low precision as they require matrix inversion or decomposition. These limitations render such methods unpopular for modern mixed-precision training. We address them by (i) formulating an inverse-free KFAC update and (ii) imposing structures in the Kronecker factors, resulting in structured inverse-free natural gradient descent (SINGD). On modern neural networks, we show that SINGD is memory-efficient and numerically robust, in contrast to KFAC, and often outperforms AdamW even in half precision. Our work closes a gap between first- and second-order methods in modern low-precision training.