2025/10/18 by Fuqun Han, Stanley Osher, Han, Fuqun +3
Computer Science · Mathematics · #FOS: Computer and information sciences #FOS: Mathematics #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques #Tensor decomposition and applications
paper · pdf · doi:10.48550/arxiv.2510.16356
openalex publication_date 2025/10/18 · openalex created_date 2025/10/22 · openalex updated_date 2026/07/28
In this work, we propose a sparse transformer architecture that incorporates prior information about the underlying data distribution directly into the transformer structure of the neural network. The design of the model is motivated by a special optimal transport problem, namely the regularized Wasserstein proximal operator, which admits a closed-form solution and turns out to be a special representation of transformer architectures. Compared with classical flow-based models, the proposed approach improves the convexity properties of the optimization problem and promotes sparsity in the generated samples. Through both theoretical analysis and numerical experiments, including applications in generative modeling and Bayesian inverse problems, we demonstrate that the sparse transformer achieves higher accuracy and faster convergence to the target distribution than classical neural ODE-based methods.