The Geometric Structure of Fully-Connected ReLU Layers

2023/10/05 by Vallin, Jonatan, Larsson, Karl, Larson, Mats G. · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2310.03482

Abstract

We formalize and interpret the geometric structure of d-dimensional fully connected ReLU layers in neural networks. The parameters of a ReLU layer induce a natural partition of the input domain, such that the ReLU layer can be significantly simplified in each sector of the partition. This leads to a geometric interpretation of a ReLU layer as a projection onto a polyhedral cone followed by an affine transformation, in line with the description in [doi:10.48550/arXiv.1905.08922] for convolutional networks with ReLU activations. Further, this structure facilitates simplified expressions for preimages of the intersection between partition sectors and hyperplanes, which is useful when describing decision boundaries in a classification setting. We investigate this in detail for a feed-forward network with one hidden ReLU-layer, where we provide results on the geometric complexity of the decision boundary generated by such networks, as well as proving that modulo an affine transformation, such a network can only generate d different decision boundaries. Finally, the effect of adding more layers to the network is discussed.

Cited by

• Marching Neurons: Accurate Surface Extraction for Neural Implicit Shapes

Related

• Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networks
• On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks
• A simple geometric proof for the benefit of depth in ReLU networks
• Theoretical Exploration of Solutions of Feedforward ReLU Networks
• The Geometry of ReLU Networks through the ReLU Transition Graph