Matricial Free Energy as a Gaussianizing Regularizer: Enhancing Autoencoders for Gaussian Code Generation

2025/10/20 by Rishi Sonthalia, Sonthalia, Rishi, Raj Rao Nadakuditi +1
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2510.17120

openalex publication_date 2025/10/20 · openalex created_date 2025/10/22 · openalex updated_date 2026/07/28

Abstract

We introduce a novel regularization scheme for autoencoders based on matricial free energy. Our approach defines a differentiable loss function in terms of the singular values of the code matrix (code dimension x batch size). From the standpoint of free probability an d random matrix theory, this loss achieves its minimum when the singular value distribution of the code matrix coincides with that of an appropriately sculpted random metric with i.i.d. Gaussian entries. Empirical simulations demonstrate that minimizing the negative matricial free energy through standard stochastic gradient-based training yields Gaussian-like codes that generalize across training and test sets. Building on this foundation, we propose a matricidal free energy maximizing autoencoder that reliably produces Gaussian codes and show its application to underdetermined inverse problems.

Citations

• Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models
• 4+3 Phases of Compute-Optimal Neural Scaling Laws
• Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalization
• Free-form Flows: Make Any Architecture a Normalizing Flow
• Spectral alignment of stochastic gradient descent for high-dimensional classification tasks
• On the Convergence Rate of Gaussianization with Random Rotations
• Double Descent and Overfitting under Noisy Inputs and Distribution Shift for Linear Denoisers
• Least Squares Regression Can Exhibit Under-Parameterized Double Descent
• EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention
• Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression
• Hessian Eigenspectra of More Realistic Nonlinear Models
• Score-Based Generative Modeling through Stochastic Differential Equations
• A Random Matrix Theory Approach to Damping in Deep Learning
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Exact expressions for double descent and implicit regularization via surrogate random design
• Generative Modeling by Estimating Gradients of the Data Distribution
• Neural Spline Flows
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Surprises in High-Dimensional Ridgeless Least Squares Interpolation
• Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
• Glow: Generative Flow with Invertible 1x1 Convolutions
• Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model
• Wasserstein Auto-Encoders
• Attention Is All You Need
• Masked Autoregressive Flow for Density Estimation
• Density estimation using Real NVP
• High-Dimensional Asymptotics of Prediction: Ridge Regression and\n Classification
• Deep Learning Face Attributes in the Wild
• NICE: Non-linear Independent Components Estimation
• The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]
• A Connection Between Score Matching and Denoising Autoencoders
• Invariant tests for multivariate normality: a critical review
• On the Empirical Distribution of Eigenvalues of a Class of Large Dimensional Random Matrices
• SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows

Related

• Gaussian AutoEncoder
• Training Autoencoders Using Stochastic Hessian-Free Optimization with LSMR
• Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods
• Rate-Distortion Auto-Encoders
• Information Potential Auto-Encoders