Geometric Cross-Modal Token Selection for Latency-Constrained Multimodal Token Communication

2026/08/04 by Joohyuk Park, Junyong Shin, Yongjeong Oh +2
Engineering · #eess.SP

paper · pdf

arxiv created 2026/08/04 · arxiv updated 2026/08/05

Abstract

This paper proposes a geometry-based joint cross-modal token selection framework for latency-constrained multimodal token communications. To capture cross-modal token dependencies, we leverage the cross-attention mechanism to project modality-specific tokens into a shared query-key space, where the modality with the fewest tokens serves as the anchor modality and the others as non-anchor modalities. Inspired by germ-grain models, we define an angular-distance metric and construct semantic grain regions around anchor queries. Based on this geometric representation, we identify cross-modal evidence shared across multiple anchor queries in this space and develop an intersection-based token selection (IBS) strategy that prioritizes non-anchor tokens whose key are covered by multiple grain regions. We further develop an erasure-aware extension, termed robust-IBS (R-IBS), for token-wise erasure channels using an expected angular-distance formulation. In both IBS and R-IBS, the grain regions are optimized for individual queries under a latency constraint, using block coordinate descent and a low-complexity greedy algorithm. Simulations corroborate the effectiveness of IBS and R-IBS under latency-constrained and token-wise erasure channels, achieving up to 31.6% and 29.2% task accuracy gains on visual question answering (VQA) and audio-visual question answering (AVQA) tasks, respectively, over existing token selection baselines.

Citations

• Joint Semantic-Channel Coding and Modulation for Token Communications
• A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity
• Text-Guided Token Communication for Wireless Image Transmission
• Token Communication in the Era of Large Models: An Information Bottleneck-Based Approach
• Low-Complexity Semantic Packet Aggregation for Token Communication via Lookahead Search
• Adaptive Semantic Token Communication for Transformer-based Edge Inference
• Task-Oriented Multimodal Token Transmission in Resource-Constrained Multiuser Networks
• Token Communications: A Large Model-Driven Framework for Cross-modal Context-aware Semantic Communications
• TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings
• Synchronous Multi-modal Semantic Communication System with Packet-level Coding
• Matryoshka Query Transformer for Large Vision-Language Models
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
• MobileVLM V2: Faster and Stronger Baseline for Vision Language Model
• Honeybee: Locality-enhanced Projector for Multimodal LLM
• Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• A survey on multimodal large language models
• Block Coordinate Descent on Smooth Manifolds: Convergence Theory and Twenty-One Examples
• Visual Instruction Tuning
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• A Unified Multi-Task Semantic Communication System for Multimodal Data
• Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
• How much training is needed in multiple-antenna wireless links?
• Multidimensional Scaling: I. Theory and Method
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding