RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer

2025/06/13 by Haotian Ni, Ni, Haotian, Hang Liu +10 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multi-Agent Systems and Negotiation #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2506.11465

openalex publication_date 2025/06/13 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

Multimodal learning faces challenges in effectively fusing information from diverse modalities, especially when modality quality varies across samples. Dynamic fusion strategies, such as attention mechanism in Transformers, aim to address such challenge by adaptively emphasizing modalities based on the characteristics of input data. However, through amounts of carefully designed experiments, we surprisingly observed that the dynamic adaptability of widely-used self-attention models diminishes. Model tends to prefer one modality regardless of data characteristics. This bias triggers a self-reinforcing cycle that progressively overemphasizes the favored modality, widening the distribution gap in attention keys across modalities and deactivating attention mechanism's dynamic properties. To revive adaptability, we propose a simple yet effective method Rolling Query (RollingQ), which balances attention allocation by rotating the query to break the self-reinforcing cycle and mitigate the key distribution gap. Extensive experiments on various multimodal scenarios validate the effectiveness of RollingQ and the restoration of cooperation dynamics is pivotal for enhancing the broader capabilities of widely deployed multimodal Transformers. The source code is available at https://github.com/GeWu-Lab/RollingQ_ICML2025.

Citations

• Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition
• MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities
• ReconBoost: Boosting Can Achieve Modality Reconcilement
• Multimodal Fusion on Low-quality Data: A Comprehensive Survey
• Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks
• Joint Multimodal Transformer for Emotion Recognition in the Wild
• A mathematical perspective on Transformers
• Enhancing multimodal cooperation via sample-level modality valuation
• Multimodal Multi-loss Fusion Network for Sentiment Analysis
• Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion
• Provable Dynamic Fusion for Low-Quality Multimodal Data
• Balanced Audiovisual Dataset for Imbalance Analysis
• Zorro: the masked multimodal transformer
• PMR: Prototypical Modal Rebalance for Multimodal Learning
• Multimodal Learning with Transformers: A Survey
• Multimodal Token Fusion for Vision Transformers
• Balanced Multimodal Learning via On-the-fly Gradient Modulation
• Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)
• Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks
• Self-attention fusion for audiovisual emotion recognition with incomplete data
• Masked Autoencoders Are Scalable Vision Learners
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Attention Bottlenecks for Multimodal Fusion
• ImageNet-21K Pretraining for the Masses
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Deep Multimodal Fusion by Channel Exchanging
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• End-to-End Object Detection with Transformers
• VGGSound: A Large-scale Audio-Visual Dataset
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• A Survey on Deep Learning for Multimodal Data Fusion
• Addressing Failure Prediction by Learning Model Confidence
• Revealing the Dark Secrets of BERT
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• What Does BERT Look At? An Analysis of BERT's Attention
• Multimodal Transformer for Unaligned Multimodal Language Sequences
• Deep Modular Co-Attention Networks for Visual Question Answering
• What Makes Training Multi-Modal Classification Networks Hard?
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• Neural Ordinary Differential Equations
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Tensor Fusion Network for Multimodal Sentiment Analysis
• Attention Is All You Need
• Multimodal Machine Learning: A Survey and Taxonomy
• Multimodal Machine Learning: A Survey and Taxonomy
• Look, Listen and Learn
• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• A Stochastic Approximation Method

Cited by

• Robust Multimodal Semantic Segmentation with Balanced Modality Contributions

Related

• MutualFormer: Multi-Modality Representation Learning via Cross-Diffusion Attention
• Deconstructing Attention: Investigating Design Principles for Effective Language Modeling
• Multimodal Transformer With a Low-Computational-Cost Guarantee
• From Multimodal to Unimodal Attention in Transformers using Knowledge\n Distillation
• Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer