Spectral Norm Regularization for Improving the Generalizability of Deep Learning

2017/05/31 by Yuichi Yoshida, Yoshida, Yuichi, Takeru Miyato +1 · 44 citations
Computer Science · Engineering · #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.1705.10941

openalex publication_date 2017/05/31 · openalex created_date 2017/06/05 · openalex updated_date 2026/07/29

Abstract

We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The Loss Surfaces of Multilayer Networks
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• Exploring the Limits of Language Modeling
• Densely Connected Convolutional Networks
• Full-Capacity Unitary Recurrent Neural Networks
• Flat Minima
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Soft-Constrained Optimization of Latent Space in Variational Autoencoders
• SineProject: Machine Unlearning for Stable Vision Language Alignment
• ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization
• Spectral-Convergent Decentralized Machine Learning: Theory and Application in Space Networks
• Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
• Position: Many generalization measures for deep learning are fragile
• Joint Discriminative-Generative Modeling via Dual Adversarial Training
• Accelerating Inference for Multilayer Neural Networks with Quantum Computers
• Reconcile Certified Robustness and Accuracy for DNN-based Smoothed Majority Vote Classifier
• Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning
• Closed-form ℓ_r norm scaling with data for overparameterized linear regression and diagonal linear networks under ℓ_p bias
• A Unified Noise-Curvature View of Loss of Trainability
• Training Invertible Linear Layers through Rank-One Perturbations
• Closing the Loop Inside Neural Networks: Causality-Guided Layer Adaptation for Fault Recovery Control
• ReachNN: Reachability Analysis of Neural-Network Controlled Systems
• Regularization for convolutional kernel tensors to avoid unstable gradient problem in convolutional neural networks
• Nearest Neighbor Projection Removal Adversarial Training
• RL Fine-Tuning Heals OOD Forgetting in SFT
• Sorting out Lipschitz function approximation
• Distance-informed Neural Processes
• S3LoRA: Safe Spectral Sharpness-Guided Pruning in Adaptation of Agent Planner
• RoMA: Robust Model Adaptation for Offline Model-based Optimization
• Efficient estimates of optimal transport via low-dimensional embeddings
• Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis
• Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product
• Training Transformers with Enforced Lipschitz Constants
• An automatic differentiation system for the age of differential privacy
• Plug and Play Splitting Techniques for Poisson Image Restoration
• Smoothed Geometry for Robust Attribution
• Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification
• Towards Efficient and Unbiased Implementation of Lipschitz Continuity in GANs
• Neural Event-Triggered Control with Optimal Scheduling
• On regularization for a convolutional kernel in neural networks
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate\n Schedule
• Approximated Orthonormal Normalisation in Training Neural Networks
• Deep Learning for Inverse Problems: Bounds and Regularizers
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Neural Abstract Reasoner
• Recovering Plasticity of Neural Networks via Soft Weight Rescaling
• Noise Stability Regularization for Improving BERT Fine-tuning
• Attributing Data for Sharpness-Aware Minimization
• LRM-1B: Towards Large Routing Model
• On Lipschitz Regularization of Convolutional Layers using Toeplitz\n Matrix Theory
• Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families

Related

• Exact Spectral Norm Regularization for Neural Networks
• A Kernel Perspective for Regularizing Deep Neural Networks
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks
• How You Start Matters for Generalization