Learning towards Minimum Hyperspherical Energy

2018/05/23 by Weiyang Liu, Liu, Weiyang, Rongmei Lin +11 · 6 citations
Computer Science · Earth and Planetary Sciences · #Advanced Neural Network Applications #Computational Physics and Python Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Geophysical and Geoelectrical Methods #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1805.09298

openalex publication_date 2018/05/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Neural networks are a powerful class of nonlinear functions that can be trained end-to-end on various applications. While the over-parametrization nature in many neural networks renders the ability to fit complex functions and the strong representation power to handle challenging tasks, it also leads to highly correlated neurons that can hurt the generalization ability and incur unnecessary computation cost. As a result, how to regularize the network to avoid undesired representation redundancy becomes an important issue. To this end, we draw inspiration from a well-known problem in physics -- Thomson problem, where one seeks to find a state that distributes N electrons on a unit sphere as evenly as possible with minimum potential energy. In light of this intuition, we reduce the redundancy regularization problem to generic energy minimization, and propose a minimum hyperspherical energy (MHE) objective as generic regularization for neural networks. We also propose a few novel variants of MHE, and provide some insights from a theoretical point of view. Finally, we apply neural networks with MHE regularization to several challenging tasks. Extensive experiments demonstrate the effectiveness of our intuition, by showing the superior performance with MHE regularization.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Minimal Riesz energy point configurations for rectifiable d -dimensional manifolds
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• CosFace: Large Margin Cosine Loss for Deep Face Recognition

Cited by

• Extreme Model Compression with Structured Sparsity at Low Precision
• Using predefined vector systems as latent space configuration for neural network supervised training on data with arbitrarily large number of classes
• Energy-Regularized Sequential Model Editing on Hyperspheres
• Spectral Scaling Laws in Language Models: How Effectively Do Feed-Forward Networks Use Their Latent Space?
• Weights-Rotated Preference Optimization for Large Language Models
• Angular Dispersion Accelerates k-Nearest Neighbors Machine Translation

Related

• Regularizing Neural Networks via Minimizing Hyperspherical Energy
• Learning with Hyperspherical Uniformity
• Eccentric Regularization: Minimizing Hyperspherical Energy without explicit projection
• Physics-enhanced Neural Networks in the Small Data Regime
• Sampling-free Inference for Ab-Initio Potential Energy Surface Networks