Multilingual Vision-Language Models, A Survey

2025/09/26 by Andrei-Alexandru Manea, Manea, Andrei-Alexandru, Jindřich Libovický +1 · 1 voice · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #cs.CL

paper · pdf · doi:10.48550/arxiv.2509.22123

arxiv published 2025/09/26 · arxiv updated 2026/05/13

Abstract

This survey examines multilingual vision-language models that process text and images across languages. We review 33 models and 23 benchmarks, spanning encoder-only and generative architectures, and identify a key tension between language neutrality (consistent cross-lingual representations) and cultural awareness (adaptation to cultural contexts). Current training methods favor neutrality through contrastive learning, while cultural awareness depends on diverse data. Two-thirds of evaluation benchmarks use translation-based approaches prioritizing semantic consistency, though recent work incorporates culturally grounded content. We find discrepancies in cross-lingual capabilities and gaps between training objectives and evaluation goals.

Citations

• T5Gemma 2: Seeing, Reading, and Understanding Longer
• Culture Affordance Atlas: Reconciling Object Diversity Through Functional Mapping
• Vision Language Models are Confused Tourists
• Traveling Across Languages: Benchmarking Cross-Lingual Consistency in Multimodal LLMs
• Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoders
• MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
• Gemma 3 Technical Report
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching
• Maya: An Instruction Finetuned Multilingual Multimodal Model
• All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages
• No Culture Left Behind: ArtELingo-28, a Benchmark of WikiArt with Captions in 28 Languages
• Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
• WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• The Llama 3 Herd of Models
• Benchmarking Vision Language Models for Cultural Understanding
• Generating Contextually-Relevant Navigation Instructions for Blind and Low Vision People
• PaliGemma: A versatile 3B VLM for transfer
• Vision-Language Models under Cultural and Inclusive Considerations
• M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks
• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
• Aya 23: Open Weight Releases to Further Multilingual Progress
• MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering
• Understanding Cross-Lingual Alignment -- A Survey
• VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments
• EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models
• Towards Measuring and Modeling "Culture" in LLMs: A Survey
• What Is Missing in Multilingual Visual Reasoning and How to Fix It
• PALO: A Polyglot Large Multimodal Model for 5B People
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
• AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
• What is "Typological Diversity" in NLP?
• MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
• SILC: Improving Vision Language Pretraining with Self-Distillation
• Improved Baselines with Visual Instruction Tuning
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• Making LLaMA SEE and Draw with SEED Tokenizer
• Demystifying CLIP Data
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization
• NLLB-CLIP -- train performant multilingual image retrieval model on a budget
• SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs
• Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
• Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models
• PaLI-X: On Scaling up a Multilingual Vision and Language Model
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Meta-learning For Vision-and-language Cross-lingual Transfer
• Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment
• Sigmoid Loss for Language Image Pre-Training
• LLaMA: Open and Efficient Foundation Language Models
• Scaling Vision Transformers to 22 Billion Parameters
• FlexiViT: One Model for All Patch Sizes
• Language Models as Agent Models
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
• MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting
• Improving the Cross-Lingual Generalisation in Visual Question Answering
• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
• BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
• Masked Vision and Language Modeling for Multi-modal Representation Learning
• No Language Left Behind: Scaling Human-Centered Machine Translation
• BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning
• Write and Paint: Generative Vision-Language Models are Unified Modal Learners
• Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training
• Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset
• Parameter-efficient Model Adaptation for Vision Transformers
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• Training language models to follow instructions with human feedback
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• Few-shot Learning with Multilingual Language Models
• MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• Visually Grounded Reasoning across Languages and Cultures
• Pix2seq: A Language Modeling Framework for Object Detection
• MURAL: Multimodal, Multitask Retrieval Across Languages
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• BEiT: BERT Pre-Training of Image Transformers
• Societal Biases in Language Generation: Progress and Challenges
• ImageNet-21K Pretraining for the Masses
• UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
• Learning Transferable Visual Models From Natural Language Supervision
• Word Alignment by Fine-tuning Embeddings on Parallel Corpora
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Transformer Feed-Forward Layers Are Key-Value Memories
• Towards Zero-shot Cross-lingual Image Retrieval
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
• Language Models are Few-Shot Learners
• On the Language Neutrality of Pre-trained Multilingual Representations
• The State and Fate of Linguistic Diversity and Inclusion in the NLP\n World
• Zero-Shot Cross-Lingual Transfer with Meta Learning
• Unsupervised Cross-lingual Representation Learning at Scale
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Root Mean Square Layer Normalization
• UNITER: UNiversal Image-TExt Representation Learning
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• From Balustrades to Pierre Vinken: Looking for Syntax in Transformer\n Self-Attentions
• Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• From Recognition to Cognition: Visual Commonsense Reasoning
• A Corpus for Reasoning About Natural Language Grounded in Photographs
• XNLI: Evaluating Cross-lingual Sentence Representations
• Lessons learned in multilingual grounded language learning
• Attention Is All You Need
• STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset
• Baselines and test data for cross-lingual inference
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
• Language Modeling with Gated Convolutional Networks
• Multi30K: Multilingual English-German Image Descriptions
• Deep Residual Learning for Image Recognition
• Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning
• A large annotated corpus for learning natural language inference
• Learning language through pictures
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• VQA: Visual Question Answering
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• CIDEr: Consensus-based Image Description Evaluation
• Microsoft COCO: Common Objects in Context
• Adaptive Mixtures of Local Experts
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Jina-VLM: Small Multilingual Vision Language Model

Discussions

• With @andrei-a-manea.bsky.social, we posted a survey on multilingual vision-language models 👉 arxiv.org/pdf/2509.22123 We reviewed 31 models+21 benchmarks. There's a tension between language neutrali [bsky, 3 points, 1 comments]

Related

• TowerVision: Understanding and Improving Multilinguality in Vision-Language Models
• Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation
• Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
• From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
• Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation