The Multilinear Structure of ReLU Networks

2017/12/29 by Laurent, Thomas, von Brecht, James · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.1712.10132

Abstract

We study the loss surface of neural networks equipped with a hinge loss criterion and ReLU or leaky ReLU nonlinearities. Any such network defines a piecewise multilinear form in parameter space. By appealing to harmonic analysis we show that all local minima of such network are non-differentiable, except for those minima that occur in a region of parameter space where the loss surface is perfectly flat. Non-differentiable minima are therefore not technicalities or pathologies; they are heart of the problem when investigating the loss of ReLU networks. As a consequence, we must employ techniques from nonsmooth analysis to study these loss surfaces. We show how to apply these techniques in some illustrative cases.

Cited by

• Constraining the outputs of ReLU neural networks

Related

• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II
• Annihilation of Spurious Minima in Two-Layer ReLU Networks
• Pure and Spurious Critical Points: a Geometric Study of Linear Networks
• Topological obstruction to the training of shallow ReLU neural networks