Data-Dependent Path Normalization in Neural Networks

2015/11/20 by Behnam Neyshabur, Neyshabur, Behnam, Ryota Tomioka +5 · 2 citations
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1511.06747

openalex publication_date 2015/11/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a unified framework for neural net normalization, regularization and optimization, which includes Path-SGD and Batch-Normalization and interpolates between them across two different dimensions. Through this framework we investigate issue of invariance of the optimization, data dependence and the connection with natural gradients.

Citations

• Natural Neural Networks
• In Search of the Real Inductive Bias: On the Role of Implicit\n Regularization in Deep Learning
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature

Cited by

• Algorithmic Regularization in Learning Deep Homogeneous Models: Layers\n are Automatically Balanced
• Information-Theoretic Framework for Understanding Modern Machine-Learning

Related

• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Weight and Gradient Centralization in Deep Neural Networks
• Path-Normalized Optimization of Recurrent Neural Networks with ReLU\n Activations
• Riemannian approach to batch normalization
• Training Deep Neural Networks Without Batch Normalization