Jeffrey Pennington

1. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
   2019/02/18 by Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz +4 · 2 voices · 12 citations
   Mathematics · Computer Science · #stat.ML #cs.LG
2. Deep Neural Networks as Gaussian Processes
   2017/11/01 by Jaehoon Lee, Yasaman Bahri, Lee, Jaehoon +9 · 63 citations
   Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Statistical Mechanics and Entropy #Target Tracking and Data Fusion in Sensor Networks
3. Training LLMs over Neurally Compressed Text
   2024/04/04 by Brian Lester, Lester, Brian, Jaehoon Lee +12 · 4 voices · 2 citations
   Computer Science · #Handwritten Text Recognition Techniques #Mathematics, Computing, and Information Processing #Natural Language Processing Techniques #cs.CL #cs.LG
4. Scaling Exponents Across Parameterizations and Optimizers
   2024/07/08 by Katie Everett, Everett, Katie, Lechao Xiao +19 · 2 voices · 20 citations
   #cs.LG
5. Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
   2023/12/11 by Avi Singh, John D. Co-Reyes, Singh, Avi +76 · 37 citations
   Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling
6. Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
   2018/06/14 by Lechao Xiao, Yasaman Bahri, Xiao, Lechao +7 · 29 citations
   Computer Science · #Generative Adversarial Networks and Image Synthesis #Advanced Neural Network Applications #Human Pose and Action Recognition
7. Sensitivity and Generalization in Neural Networks: an Empirical Study
   2018/02/23 by Roman Novak, Yasaman Bahri, Novak, Roman +7 · 20 citations
   Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE)
8. Small-scale proxies for large-scale Transformer training instabilities
   2023/09/25 by Mitchell Wortsman, Peter J. Liu, Wortsman, Mitchell +29 · 27 citations
   Computer Science · Decision Sciences · #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Forecasting Techniques and Applications #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG)
9. Bayesian Deep Convolutional Networks with Many Channels are Gaussian\n Processes
   2018/10/11 by Roman Novak, Novak, Roman, Lechao Xiao +16 · 22 citations
   Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Neural and Evolutionary Computing (cs.NE)
10. 4+3 Phases of Compute-Optimal Neural Scaling Laws
    2024/05/23 by Elliot Paquette, Paquette, Elliot, Courtney Paquette +5 · 21 citations
    Computer Science · #Neural Networks and Applications
11. The Emergence of Spectral Universality in Deep Networks
    2018/02/27 by Jeffrey Pennington, Samuel S. Schoenholz, Pennington, Jeffrey +3 · 12 citations
    Computer Science · Engineering · #Blind Source Separation Techniques #Image and Signal Denoising Methods #Sparse and Compressive Sensing Techniques
12. A Correspondence Between Random Neural Networks and Statistical Field\n Theory
    2017/10/17 by Samuel S. Schoenholz, Schoenholz, Samuel S., Jeffrey Pennington +3 · 5 citations
    Computer Science · Physics and Astronomy · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Physical sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Statistical Mechanics and Entropy
13. Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
    2020/01/16 by Wei Hu, Lechao Xiao, Hu, Wei +3 · 6 citations
    Computer Science · #Speech Recognition and Synthesis #Neural Networks and Applications #Gaussian Processes and Bayesian Inference
14. Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks
    2025/07/02 by Shikai Qiu, Lechao Xiao, Qiu, Shikai +7 · 3 voices · 11 citations
    #cs.LG
15. Understanding Double Descent Requires a Fine-Grained Bias-Variance\n Decomposition
    2020/11/04 by Ben Adlam, Adlam, Ben, Jeffrey Pennington +1 · 5 citations
    Computer Science · Materials Science · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Machine Learning in Materials Science #Stochastic Gradient Optimization Techniques
16. Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions
    2022/06/15 by Courtney Paquette, Elliot Paquette, Paquette, Courtney +5 · 4 citations
    Biochemistry, Genetics and Molecular Biology · Computer Science · Mathematics · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Optimization and Control (math.OC) #Probability (math.PR) #Single-cell and spatial transcriptomics #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques
17. Second-order regression models exhibit progressive sharpening to the edge of stability
    2022/10/10 by Atish Agarwala, Fabián Pedregosa, Agarwala, Atish +3 · 3 citations
    Computer Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques
18. The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
    2020/06/25 by Wei Hu, Lechao Xiao, Hu, Wei +5 · 2 citations
    Computer Science · #Stochastic Gradient Optimization Techniques #Machine Learning and Algorithms #Neural Networks and Applications
19. A Strong Gravitational Lens Is Worth a Thousand Dark Matter Halos: Inference on Small-Scale Structure Using Sequential Methods
    2024/04/22 by Sebastian Wagner-Carena, Wagner-Carena, Sebastian, Jae‐Hoon Lee +9 · 3 citations
    Physics and Astronomy · #Cosmology and Gravitation Theories #Cosmology and Nongalactic Astrophysics (astro-ph.CO) #FOS: Physical sciences #Pulsars and Gravitational Waves Research #Scientific Research and Discoveries
20. Bootstrapping six-gluon scattering in planar \cal N=4 super-Yang-Mills theory
    2014/07/17 by Lance J. Dixon, J. M. Drummond, Dixon, Lance J. +7 · 1 citation
    Physics and Astronomy · #Black Holes and Theoretical Physics #FOS: Physical sciences #High Energy Physics - Theory (hep-th) #Particle physics theoretical and experimental studies #Quantum Chromodynamics and Particle Interactions
21. A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning
    2019/12/02 by Ben Adlam, Jake Levinson, Adlam, Ben +3 · 2 citations
    Computer Science · Mathematics · #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Random Matrices and Applications #Stochastic Gradient Optimization Techniques
22. Disentangling Trainability and Generalization in Deep Neural Networks
    2019/12/30 by Xiao, Lechao, Jeffrey Pennington, Samuel S. Schoenholz +2 · 2 citations
    Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques
23. Covariate Shift in High-Dimensional Random Feature Regression
    2021/11/16 by Nilesh Tripuraneni, Ben Adlam, Tripuraneni, Nilesh +3 · 1 citation
    Computer Science · #Domain Adaptation and Few-Shot Learning #Machine Learning and Algorithms #Stochastic Gradient Optimization Techniques