Who Said Neural Networks Aren't Linear?

2025/10/09 by Nimrod Berman, Berman, Nimrod, Assaf Hallak +3 · 3 voices
Computer Science · Materials Science · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Materials Science #Model Reduction and Neural Networks #Stochastic Gradient Optimization Techniques #cs.LG

paper · pdf · doi:10.48550/arxiv.2510.08570

openalex publication_date 2025/10/09 · arxiv published 2025/10/09 · openalex created_date 2025/10/11 · arxiv updated 2026/02/20 · openalex updated_date 2026/07/28

Abstract

Neural networks are famously nonlinear. However, linearity is defined relative to a pair of vector spaces, f:X → Y. Leveraging the algebraic concept of transport of structure, we propose a method to explicitly identify non-standard vector spaces where a neural network acts as a linear operator. When sandwiching a linear operator A between two invertible neural networks, f(x)=g_y^-1(A g_x(x)), the corresponding vector spaces X and Y are induced by newly defined addition and scaling actions derived from g_x and g_y. We term this kind of architecture a Linearizer. This framework makes the entire arsenal of linear algebra, including SVD, pseudo-inverse, orthogonal projection and more, applicable to nonlinear mappings. Furthermore, we show that the composition of two Linearizers that share a neural network is also a Linearizer. We leverage this property and demonstrate that training diffusion models using our architecture makes the hundreds of sampling steps collapse into a single step. We further utilize our framework to enforce idempotency (i.e. f(f(x))=f(x)) on networks leading to a globally projective generative model and to demonstrate modular style transfer.

Citations

• One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling
• IT³: Idempotent Test-Time Training
• Idempotent Generative Network
• Representation Learning via Manifold Flattening and Reconstruction
• Consistency Models
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• Flow Matching for Generative Modeling
• Elucidating the Design Space of Diffusion-Based Generative Models
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
• Diffusion Models Beat GANs on Image Synthesis
• Score-Based Generative Modeling through Stochastic Differential Equations
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• Learning Group Structure and Disentangled Representations of Dynamical Environments
• Glow: Generative Flow with Invertible 1x1 Convolutions
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Density Estimation Using Real NVP
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• On the Expressive Power of Deep Learning: A Tensor Analysis
• Variational Inference with Normalizing Flows
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Deep Learning Face Attributes in the Wild
• NICE: Non-linear Independent Components Estimation
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Estimating or Propagating Gradients Through Stochastic Neurons
• Spectral Properties of Dynamical Systems, Model Reduction and Decompositions
• Multilayer feedforward networks are universal approximators
• Gradient-based learning applied to document recognition

Discussions

• Who Said Neural Networks Aren't Linear? [hn, 2 points, 0 comments]
• This is the most interesting paper I've seen this year so far: arxiv.org/pdf/2510.08570 [bsky, 1 points, 0 comments]
• ..discover that complex systems might have simple mathematical hearts beating underneath. Read more: https://arxiv.org/abs/2510.08570v1 (5/5) [bsky, 0 points, 0 comments]

Related

• Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models
• A Tutorial on Neural Networks and Gradient-free Training
• Training Linear Neural Networks: Non-Local Convergence and Complexity Results
• On Non-Linear operators for Geometric Deep Learning
• Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning