On the Geometry of Deep Learning

2024/08/09 by Randall Balestriero, Ahmed Imtiaz Humayun, Balestriero, Randall +3 · 1 voice
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2408.04809

arxiv published 2024/08/09 · arxiv updated 2025/01/14

Abstract

In this paper, we overview one promising avenue of progress at the mathematical foundation of deep learning: the connection between deep networks and function approximation by affine splines (continuous piecewise linear functions in multiple dimensions). In particular, we will overview work over the past decade on understanding certain geometrical properties of a deep network's affine spline mapping, in particular how it tessellates its input space. As we will see, the affine spline connection and geometrical viewpoint provide a powerful portal through which to view, analyze, and improve the inner workings of a deep network.

Discussions

• The findings stem from expressing SAEs as splines (arxiv.org/abs/2408.04809) and doing a deep dive into their partition, constraints, and underlying geometry! We not only characterize their input spac [bsky, 0 points, 1 comments]

Related

• Mad Max: Affine Spline Insights into Deep Learning
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Mathematical Foundations of Geometric Deep Learning
• The Geometry of Deep Networks: Power Diagram Subdivision
• The algebra and the geometry aspect of Deep learning