auto-fpt: Automating Free Probability Theory Calculations for Machine Learning Theory

2025/04/14 by Arjun Subramonian, Subramonian, Arjun, Elvis Dohmatob +1
Mathematics · Computer Science · #Random Matrices and Applications #Stochastic Gradient Optimization Techniques #Quantum Computing Algorithms and Architecture

paper · pdf · doi:10.48550/arxiv.2504.10754

Abstract

A large part of modern machine learning theory often involves computing the high-dimensional expected trace of a rational expression of large rectangular random matrices. To symbolically compute such quantities using free probability theory, we introduce auto-fpt, a lightweight Python and SymPy-based tool that can automatically produce a reduced system of fixed-point equations which can be solved for the quantities of interest, and effectively constitutes a theory. We overview the algorithmic ideas underlying auto-fpt and its applications to various interesting problems, such as the high-dimensional error of linearized feed-forward neural networks, recovering well-known results. We hope that auto-fpt streamlines the majority of calculations involved in high-dimensional analysis, while helping the machine learning community reproduce known and uncover new phenomena.

Citations

• Strong Model Collapse
• An Effective Theory of Bias Amplification
• Linear matrix pencils and noncommutative convexity
• High-dimensional analysis of double descent for linear regression with random projections
• Demystifying Disagreement-on-the-Line in High Dimensions
• Robust Linear Regression: Gradient-descent, Early-stopping, and Beyond
• Fundamental tradeoffs between memorization and robustness in random features and neural tangent regimes
• Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition
• The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
• Array Programming with NumPy
• Asymptotics of Ridge (less) Regression under General Source Condition
• Surprises in High-Dimensional Ridgeless Least Squares Interpolation
• Reconciling modern machine-learning practice and the classical bias–variance trade-off
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• Applications of Realizations (aka Linearizations) to Free Probability
• High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification
• Subordination for the sum of two random matrices
• A Deterministic Equivalent for the Analysis of Correlated MIMO Multiple Access Channels
• DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES

Related

• Learning Deterministic Weighted Automata with Queries and Counterexamples
• A Fast Convoluted Story: Scaling Probabilistic Inference for Integer Arithmetic
• AutoEP: LLMs-Driven Automation of Hyperparameter Evolution for Metaheuristic Algorithms
• A Study of the Mathematics of Deep Learning
• Probabilistic Arithmetic Automata and their Applications