Neural Stein critics with staged L²-regularization

2022/07/07 by Matthew Repasky, Repasky, Matthew, Xiuyuan Cheng +3 · 1 citation
Computer Science · Physics and Astronomy · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #FOS: Mathematics #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Statistics Theory (math.ST)

paper · pdf · doi:10.48550/arxiv.2207.03406

openalex publication_date 2022/07/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning to differentiate model distributions from observed data is a fundamental problem in statistics and machine learning, and high-dimensional data remains a challenging setting for such problems. Metrics that quantify the disparity in probability distributions, such as the Stein discrepancy, play an important role in high-dimensional statistical testing. In this paper, we investigate the role of L² regularization in training a neural network Stein critic so as to distinguish between data sampled from an unknown probability distribution and a nominal model distribution. Making a connection to the Neural Tangent Kernel (NTK) theory, we develop a novel staging procedure for the weight of regularization over training time, which leverages the advantages of highly-regularized training at early times. Theoretically, we prove the approximation of the training dynamic by the kernel optimization, namely the ``lazy training'', when the L² regularization weight is large, and training on n samples converge at a rate of O(n^-1/2) up to a log factor. The result guarantees learning the optimal critic assuming sufficient alignment with the leading eigen-modes of the zero-time NTK. The benefit of the staged L² regularization is demonstrated on simulated high dimensional data and an application to evaluating generative models of image data.

Cited by

• Boosting Statistic Learning with Synthetic Data from Pretrained Large Models

Related

• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling