2025/06/28 by Ziqi Zhong, Zhong, Ziqi, Xunzhu Tang +1
Computer Science · #Abstraction #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Context (archaeology) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Key (lock) #Modalities #Multimodal Machine Learning Applications #Normalization (sociology) #Question answering #Redundancy (engineering) #Representation (politics) #Security token #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2507.00068
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/06/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
While multi-modal learning has advanced significantly, current approaches often treat modalities separately, creating inconsistencies in representation and reasoning. We introduce MANTA (Multi-modal Abstraction and Normalization via Textual Alignment), a theoretically-grounded framework that unifies visual and auditory inputs into a structured textual space for seamless processing with large language models. MANTA addresses four key challenges: (1) semantic alignment across modalities with information-theoretic optimization, (2) adaptive temporal synchronization for varying information densities, (3) hierarchical content representation for multi-scale understanding, and (4) context-aware retrieval of sparse information from long sequences. We formalize our approach within a rigorous mathematical framework, proving its optimality for context selection under token constraints. Extensive experiments on the challenging task of Long Video Question Answering show that MANTA improves state-of-the-art models by up to 22.6% in overall accuracy, with particularly significant gains (27.3%) on videos exceeding 30 minutes. Additionally, we demonstrate MANTA's superiority on temporal reasoning tasks (23.8% improvement) and cross-modal understanding (25.1% improvement). Our framework introduces novel density estimation techniques for redundancy minimization while preserving rare signals, establishing new foundations for unifying multimodal representations through structured text.