Tuning Frequency Bias in Neural Network Training with Nonuniform Data

2022/05/28 by Yu, Annan, Yang, Yunan, Townsend, Alex
#68Q32 #68T07 #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2205.14300

Abstract

Small generalization errors of over-parameterized neural networks (NNs) can be partially explained by the frequency biasing phenomenon, where gradient-based algorithms minimize the low-frequency misfit before reducing the high-frequency residuals. Using the Neural Tangent Kernel (NTK), one can provide a theoretically rigorous analysis for training where data are drawn from constant or piecewise-constant probability densities. Since most training data sets are not drawn from such distributions, we use the NTK model and a data-dependent quadrature rule to theoretically quantify the frequency biasing of NN training given fully nonuniform data. By replacing the loss function with a carefully selected Sobolev norm, we can further amplify, dampen, counterbalance, or reverse the intrinsic frequency biasing in NN training.

Related

• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• Understanding the dynamics of the frequency bias in neural networks
• A Revision of Neural Tangent Kernel-based Approaches for Neural Networks
• On the Inductive Bias of Neural Tangent Kernels
• Just One Layer Norm Guarantees Stable Extrapolation