Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functions

2022/09/29 by Jacot, Arthur · 2 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2209.15055

Abstract

We show that the representation cost of fully connected neural networks with homogeneous nonlinearities - which describes the implicit bias in function space of networks with L₂-regularization or with losses such as the cross-entropy - converges as the depth of the network goes to infinity to a notion of rank over nonlinear functions. We then inquire under which conditions the global minima of the loss recover the `true' rank of the data: we show that for too large depths the global minimum will be approximately rank 1 (underestimating the rank); we then argue that there is a range of depths which grows with the number of datapoints where the true rank is recovered. Finally, we discuss the effect of the rank of a classifier on the topology of the resulting class boundaries and show that autoencoders with optimal nonlinear rank are naturally denoising.

Cited by

• Deep Learning as a Convex Paradigm of Computation: Minimizing Circuit Size with ResNets
• Conv4Rec: A 1-by-1 Convolutional AutoEncoder for User Profiling through Joint Analysis of Implicit and Explicit Feedbacks

Related

• Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L²-Regularized Nonlinear Networks
• The Low-Rank Simplicity Bias in Deep Networks
• Theory IIIb: Generalization in Deep Networks
• Implicit bias of SGD in L₂-regularized linear DNNs: One-way jumps from high to low rank
• The Persistence of Neural Collapse Despite Low-Rank Bias