Neural Networks Should Be Wide Enough to Learn Disconnected Decision\n Regions

2018/02/28 by Quynh Nguyen, Nguyen, Quynh, Mahesh Chandra Mukkamala +3 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.1803.00094

openalex publication_date 2018/02/28 · openalex created_date 2022/10/02 · openalex updated_date 2026/07/28

Abstract

In the recent literature the important role of depth in deep learning has\nbeen emphasized. In this paper we argue that sufficient width of a feedforward\nnetwork is equally important by answering the simple question under which\nconditions the decision regions of a neural network are connected. It turns out\nthat for a class of activation functions including leaky ReLU, neural networks\nhaving a pyramidal structure, that is no layer has more hidden units than the\ninput dimension, produce necessarily connected decision regions. This implies\nthat a sufficiently wide hidden layer is necessary to guarantee that the\nnetwork can produce disconnected decision regions. We discuss the implications\nof this result for the construction of neural networks, in particular the\nrelation to the problem of adversarial manipulation of classifiers.\n

Citations

• Sum-Product Networks: A New Deep Architecture
• Adversarial Machine Learning at Scale
• Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
• The loss surface of deep and wide neural networks
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• A Tropical Approach to Neural Networks with Piecewise Linear Activations
• Caffe: Convolutional Architecture for Fast Feature Embedding

Cited by

• Runtime Analysis of Evolutionary NAS for Multiclass Classification
• Topological Deep Learning: Classification Neural Networks
• Understanding Nonlinear Implicit Bias via Region Counts in Input Space

Related

• When Can Neural Networks Learn Connected Decision Regions?
• On decision regions of narrow deep neural networks
• The Expressive Power of Neural Networks: A View from the Width
• Width is Less Important than Depth in ReLU Neural Networks
• Classification regions of deep neural networks