2022/10/10 by Sho Yaida, Yaida, Sho
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #FOS: Physical sciences #High Energy Physics - Theory (hep-th) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Model Reduction and Neural Networks #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2210.04909
openalex publication_date 2022/10/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In this note, we first derive a one-parameter family of hyperparameter scaling strategies that interpolates between the neural-tangent scaling and mean-field/maximal-update scaling. We then calculate the scalings of dynamical observables -- network outputs, neural tangent kernels, and differentials of neural tangent kernels -- for wide and deep neural networks. These calculations in turn reveal a proper way to scale depth with width such that resultant large-scale models maintain their representation-learning ability. Finally, we observe that various infinite-width limits examined in the literature correspond to the distinct corners of the interconnected web spanned by effective theories for finite-width neural networks, with their training dynamics ranging from being weakly-coupled to being strongly-coupled.