Topological Alignment of Shared Vision-Language Embedding Space

2025/10/13 by You, Junwon, Kang, Dasol, Jung, Jae-Hun
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.10889

Abstract

Contrastive Vision-Language Models (VLMs) have demonstrated strong zero-shot capabilities. However, their cross-modal alignment remains biased toward English due to limited multilingual multimodal data. Recent multilingual extensions have alleviated this gap but enforce instance-level alignment while neglecting the global geometry of the shared embedding space. We address this problem by introducing ToMCLIP (Topological Alignment for Multilingual CLIP), a topology-aware framework aligning embedding spaces with topology-preserving constraints. The proposed method applies persistent homology to define a topological alignment loss and approximates persistence diagram with theoretical error bounds using graph sparsification strategy. This work validates the proposed approach, showing enhanced structural coherence of multilingual representations, higher zero-shot accuracy on the CIFAR-100, and stronger multilingual retrieval performance on the xFlickr&CO. Beyond VLMs, the proposed approach provides a general method for incorporating topological alignment into representation learning.

Citations

• Topology-Guided Knowledge Distillation for Efficient Point Cloud Processing
• Topology-Aware CLIP Few-Shot Learning
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Gemma 2: Improving Open Language Models at a Practical Size
• PaliGemma: A versatile 3B VLM for transfer
• Topological Perspectives on Optimal Multimodal Embedding Spaces
• An Introduction to Vision-Language Modeling
• Yi: Open Foundation Models by 01.AI
• Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning
• Tensor-view Topological Graph Neural Network
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• Improved Baselines with Visual Instruction Tuning
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• Visual Instruction Tuning
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Scaling Language-Image Pre-training via Masking
• Flamingo: a Visual Language Model for Few-Shot Learning
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• Learning Transferable Visual Models From Natural Language Supervision
• Wasserstein Stability for Persistence Diagrams
• Unsupervised Cross-lingual Representation Learning at Scale
• Topological Autoencoders
• PersLay: A Neural Network Layer for Persistence Diagrams and New Graph Topological Signatures
• On the shortest spanning subtree of a graph and the traveling salesman problem

Related

• Do Vision and Language Encoders Represent the World Similarly?
• Topology-Aware CLIP Few-Shot Learning
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding
• uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data
• HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment