Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizer

2026/01/21 by Yanan Li, Christina Yi Jin, Yuan Jin +5 · 1 voice
Biochemistry, Genetics and Molecular Biology · Computer Science · #q-bio.GN #cs.CL

paper · pdf

arxiv published 2026/01/21 · arxiv updated 2026/05/08

Abstract

A central challenge in developing Multimodal Large Language Models (MLLMs) is effectively integrating heterogeneous inputs into a cohesive reasoning engine. Current paradigms predominantly rely on modular architectures that introduce modality-specific encoders and cross-modal fusion mechanisms. However, these designs are fundamentally bottlenecked by a geometric modality gap, forcing the LLM to expend significant computational capacity on geometric reconciliation rather than deep cross-modal reasoning. In this work, we formally characterize this modality gap and theoretically demonstrate that native architectures, specifically those employing a unified vocabulary, intrinsically maintain a zero-gap state across all hidden layers. Guided by these theoretical findings, we propose One Tokenizer, a native architecture that maps all modalities directly into a shared token space. We empirically validate this framework on a DNA--text multimodal testbed. Our extensive evaluations reveal that by achieving seamless integration within the LLM's native latent space, One Tokenizer consistently outperforms encoder-based modular counterparts, providing a fundamentally superior framework for deep biological reasoning.

Citations

• GeneChat: A Multi-Modal Large Language Model for Gene Function Prediction
• BioReason: Incentivizing Multimodal Biological Reasoning within a DNA-LLM Model
• Qwen3 Technical Report
• Kimi-VL Technical Report
• Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
• Emu3: Next-Token Prediction is All You Need
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
• Unveiling Encoder-Free Vision-Language Models
• A Concept-Based Explainability Framework for Large Multimodal Models
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
• Chameleon: Mixed-Modal Early-Fusion Foundation Models
• Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
• MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
• Improved Baselines with Visual Instruction Tuning
• Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP
• Multimodal Neurons in Pretrained Text-Only Transformers
• A survey on multimodal large language models
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Visual Instruction Tuning
• MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation
• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• Learning Transferable Visual Models From Natural Language Supervision
• The Lipschitz Constant of Self-Attention
• Layer Normalization
• Neural Machine Translation of Rare Words with Subword Units

Discussions

• Yanan Li, Christina Yi Jin, Yuan Jin, Manli Luo, Tie Xu, Shuai Jiao, Wei He, Qing Zhang: Alignment or Integration? Rethinking Multimodal Fusion in DNA-language Foundation Models https://arxiv.org/abs/ [bsky, 1 points, 0 comments]

Related

• Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
• TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
• FysicsWorld: A Unified Full-Modality Benchmark for Any-to-Any Understanding, Generation, and Reasoning
• Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark