Quantum Doubly Stochastic Transformers

2025/04/22 by Jannis Born, Born, Jannis, Filip Skogh +9 · 1 citation
Computer Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #Computational Engineering #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Finance #Machine Learning (cs.LG) #Quantum Computing Algorithms and Architecture #Quantum and electron transport phenomena #Quantum optics and atomic interactions #and Science (cs.CE)

paper · pdf · doi:10.48550/arxiv.2504.16275

openalex publication_date 2025/04/22 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

At the core of the Transformer, the softmax normalizes the attention matrix to be right stochastic. Previous research has shown that this often de-stabilizes training and that enforcing the attention matrix to be doubly stochastic (through Sinkhorn's algorithm) consistently improves performance across different tasks, domains and Transformer flavors. However, Sinkhorn's algorithm is iterative, approximative, non-parametric and thus inflexible w.r.t. the obtained doubly stochastic matrix (DSM). Recently, it has been proven that DSMs can be obtained with a parametric quantum circuit, yielding a novel quantum inductive bias for DSMs with no known classical analogue. Motivated by this, we demonstrate the feasibility of a hybrid classical-quantum doubly stochastic Transformer (QDSFormer) that replaces the softmax in the self-attention layer with a variational quantum circuit. We study the expressive power of the circuit and find that it yields more diverse DSMs that better preserve information than classical operators. Across multiple small-scale object recognition tasks, we find that our QDSFormer consistently surpasses both a standard ViT and other doubly stochastic Transformers. Beyond the Sinkformer, this comparison includes a novel quantum-inspired doubly stochastic Transformer (based on QR decomposition) that can be of independent interest. Our QDSFormer also shows improved training stability and lower performance variation suggesting that it may mitigate the notoriously unstable training of ViTs on small-scale data.

Citations

• LOTFormer: Doubly-Stochastic Linear Attention via Low-Rank Optimal Transport
• Variational quantum algorithms for permutation-based combinatorial problems: Optimal ansatz generation with applications to quadratic assignment problems and beyond
• ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Plans
• Towards Better Multi-head Attention via Channel-wise Sample Permutation
• The Llama 3 Herd of Models
• Unraveling Molecular Structure: A Multimodal Spectroscopic Dataset for Chemistry
• Quantum Implicit Neural Representations
• Quixer: A Quantum Transformer Model
• QClusformer: A Quantum Transformer-based Framework for Unsupervised Visual Clustering
• Quantum Positional Encodings for Graph Neural Networks
• Quantum computing with Qiskit
• Accurate structure prediction of biomolecular interactions with AlphaFold 3
• Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention
• OTSeg: Multi-prompt Sinkhorn Attention for Zero-Shot Semantic Segmentation
• Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems
• Towards quantum-enabled cell-centric therapeutics
• The emergence of clusters in self-attention dynamics
• Micrograph segmentations for DDEVD
• Segment Anything
• Stabilizing Transformer Training by Preventing Attention Entropy Collapse
• A Study on ReLU and Softmax in Transformer
• Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets
• Quantum Vision Transformers
• Rethinking Initialization of the Sinkhorn Algorithm
• Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
• Sinkformers: Transformers with Doubly Stochastic Attention
• Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanism
• Best Approximate Quantum Compiling Problems
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• Doubly Stochastic Subspace Clustering
• Power of data in quantum machine learning
• The power of quantum neural networks
• MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A rigorous and robust quantum speed-up in supervised machine learning
• A gradient descent perspective on Sinkhorn
• Optimal Approximation of Doubly Stochastic Matrices
• Simulating quantum many-body dynamics on a current digital quantum computer
• Quantum-assisted quantum compiling
• Supervised learning with quantum enhanced feature spaces
• Barren plateaus in quantum neural network training landscapes
• OSQP: An Operator Splitting Solver for Quadratic Programs
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Attention Is All You Need
• Noise tailoring for scalable quantum computation via randomized compiling
• Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances
• Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions
• Volume of the set of unistochastic matrices of order 3 and the mean Jarlskog invariant
• On the volume of the polytope of doubly stochastic matrices
• A Relationship Between Arbitrary Positive Matrices and Doubly Stochastic Matrices

Cited by

• LOTFormer: Doubly-Stochastic Linear Attention via Low-Rank Optimal Transport

Related

• Quantum Vision Transformers
• Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention
• Quantum Adaptive Self-Attention for Quantum Transformer Models
• HQViT: Hybrid Quantum Vision Transformer for Image Classification
• A Survey of Quantum Transformers: Architectures, Challenges and Outlooks