On Regularizing Coordinate-MLPs

2022/02/01 by Sameera Ramasinghe, Ramasinghe, Sameera, Lachlan Ewen MacDonald +3
Engineering · Computer Science · Physics and Astronomy · #Structural Health Monitoring Techniques #Neural Networks and Applications #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2202.00790

Abstract

We show that typical implicit regularization assumptions for deep neural networks (for regression) do not hold for coordinate-MLPs, a family of MLPs that are now ubiquitous in computer vision for representing high-frequency signals. Lack of such implicit bias disrupts smooth interpolations between training samples, and hampers generalizing across signal regions with different spectra. We investigate this behavior through a Fourier lens and uncover that as the bandwidth of a coordinate-MLP is enhanced, lower frequencies tend to get suppressed unless a suitable prior is provided explicitly. Based on these insights, we propose a simple regularization technique that can mitigate the above problem, which can be incorporated into existing networks without any architectural modifications.

Related

• Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks
• Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks
• Frequency Regularization: Restricting Information Redundancy of Convolutional Neural Networks