Representation Potentials of Foundation Models for Multimodal Alignment: A Survey

2025/10/05 by Jianglin Lu, Lu, Jianglin, Hailing Wang +9 · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Natural Language Processing Techniques #Semantic Web and Ontologies #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2510.05184

openalex publication_date 2025/10/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Foundation models learn highly transferable representations through large-scale pretraining on diverse data. An increasing body of research indicates that these representations exhibit a remarkable degree of similarity across architectures and modalities. In this survey, we investigate the representation potentials of foundation models, defined as the latent capacity of their learned representations to capture task-specific information within a single modality while also providing a transferable basis for alignment and unification across modalities. We begin by reviewing representative foundation models and the key metrics that make alignment measurable. We then synthesize empirical evidence of representation potentials from studies in vision, language, speech, multimodality, and neuroscience. The evidence suggests that foundation models often exhibit structural regularities and semantic consistencies in their representation spaces, positioning them as strong candidates for cross-modal transfer and alignment. We further analyze the key factors that foster representation potentials, discuss open questions, and highlight potential challenges.

Citations

• Disentangling the Factors of Convergence between Brains and Computer Vision Models
• Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0
• Update Your Transformer to the Latest Release: Re-Basin of Task Vectors
• Shared Global and Local Geometry of Language Model Embeddings
• Activation Space Interventions Can Be Transferred Between Large Language Models
• Brain-Model Evaluations Need the NeuroAI Turing Test
• Dual Diffusion for Unified Image Generation and Understanding
• How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations
• What Representational Similarity Measures Imply about Decodable Information
• Objective drives the consistency of representational similarity across datasets
• DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
• What Do Speech Foundation Models Not Learn About Speech?
• Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures
• Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders
• How Redundant Is the Transformer Stack in Speech Representation Models?
• The Llama 3 Herd of Models
• Truth is Universal: Robust Detection of Lies in LLMs
• Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity
• A Comprehensive Survey of Foundation Models in Medicine
• A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning
• Multilingual Diversity Improves Vision-Language Representations
• Emergence of a High-Dimensional Abstraction Phase in Language Transformers
• Instruction Diversity Drives Generalization To Unseen Tasks
• Universal Neurons in GPT2 Language Models
• Do Vision and Language Encoders Represent the World Similarly?
• A Vision Check-up for Language Models
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Understanding the Inner Workings of Language Models Through Representation Dissimilarity
• Do self-supervised speech and language models extract similar representations as human brain?
• Large language models converge toward human-like concept organization
• Can Language Models Learn to Listen?
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• What Do Self-Supervised Speech Models Know About Words?
• Rosetta Neurons: Mining the Common Units in a Model Zoo
• A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
• Similarity of Neural Network Models: A Survey of Functional and Representational Measures
• ZipIt! Merging Models from Different Tasks without Training
• Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representations
• Grounding Language Models to Images for Multimodal Inputs and Outputs
• ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders
• Robust Speech Recognition via Large-Scale Weak Supervision
• Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions
• Comparative layer-wise analysis of self-supervised speech models
• Reliability of CKA as a Similarity Measure in Deep Learning
• Relative representations enable zero-shot latent space communication
• Linearly Mapping from Image to Text Space
• PaLI: A Jointly-Scaled Multilingual Language-Image Model
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?
• Similarity and Matching of Neural Network Representations
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Inductive Biases and Variable Creation in Self-Attention Mechanisms
• Do Self-Supervised and Supervised Methods Learn Similar Visual Representations?
• Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers
• Do Vision Transformers See Like Convolutional Neural Networks?
• On the Opportunities and Risks of Foundation Models
• Revisiting Model Stitching to Compare Neural Representations
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Emerging Properties in Self-Supervised Vision Transformers
• Learning Transferable Visual Models From Natural Language Supervision
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Similarity Analysis of Self-Supervised Speech Representations
• On Linear Identifiability of Learned Representations
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Similarity of Neural Network Representations Revisited
• Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation
• Insights on representational similarity in neural networks with canonical correlation
• World Models
• Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems
• Attention Is All You Need
• Deep Residual Learning for Image Recognition
• Convergent Learning: Do different neural networks learn the same representations?
• Understanding image representations by measuring their equivariance and equivalence
• Supervised Feature Selection via Dependence Estimation
• Relational reasoning and inductive bias in transformers and large language models
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Restore-R1: Efficient Image Restoration Agents via Reinforcement Learning with Multimodal LLM Perceptual Feedback
• Platonic Representations in the Human Brain: Unsupervised Recovery of Universal Geometry
• Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
• When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling

Related

• Understanding the Emergence of Multimodal Representation Alignment
• Escaping Plato's Cave: JAM for Aligning Independently Trained Vision and Language Models
• A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models
• FLAVA: A Foundational Language And Vision Alignment Model
• How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations