Any Deep ReLU Network is Shallow

2023/06/20 by Mattia Jacopo Villani, Nandi Schoots, Villani, Mattia Jacopo +1 · 3 voices
Computer Science · #Blind Source Separation Techniques #Machine Learning and Algorithms #Neural Networks and Applications #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2306.11827

openalex publication_date 2023/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We constructively prove that every deep ReLU network can be rewritten as a functionally identical three-layer network with weights valued in the extended reals. Based on this proof, we provide an algorithm that, given a deep ReLU network, finds the explicit weights of the corresponding shallow network. The resulting shallow network is transparent and used to generate explanations of the model s behaviour.

Discussions

• Any Deep ReLU Network Is Shallow [hn, 134 points, 58 comments]
• Any Deep ReLU Network Is Shallow [hn, 18 points, 5 comments]
• Any Deep ReLU Network is Shallow [lobsters, 7 points, 1 comments]

Related

• Deep Equals Shallow for ReLU Networks in Kernel Regimes
• Width is Less Important than Depth in ReLU Neural Networks
• Deep vs. shallow networks : An approximation theory perspective
• Unwrapping All ReLU Networks
• Deep ReLU Networks Have Surprisingly Simple Polytopes