2021/01/18 by Asaf Amrami, Yoav Goldberg, Amrami, Asaf +1
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2101.07126
openalex publication_date 2021/01/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We present a simple proof for the benefit of depth in multi-layer feedforward network with rectified activation ("depth separation"). Specifically we present a sequence of classification problems indexed by m such that (a) for any fixed depth rectified network there exist an m above which classifying problem m correctly requires exponential number of parameters (in m); and (b) for any problem in the sequence, we present a concrete neural network with linear depth (in m) and small constant width (≤ 4) that classifies the problem with zero error. The constructive proof is based on geometric arguments and a space folding construction. While stronger bounds and results exist, our proof uses substantially simpler tools and techniques, and should be accessible to undergraduate students in computer science and people with similar backgrounds.