Does Data Scaling Lead to Visual Compositional Generalization?

2025/07/09 by Arnas Uselis, Uselis, Arnas, Andrea Dittadi +3 · 4 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Geochemistry and Geologic Mapping #Machine Learning (cs.LG) #Topological and Geometric Data Analysis

paper · pdf · doi:10.48550/arxiv.2507.07102

openalex publication_date 2025/07/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Compositional understanding is crucial for human intelligence, yet it remains unclear whether contemporary vision models exhibit it. The dominant machine learning paradigm is built on the premise that scaling data and model sizes will improve out-of-distribution performance, including compositional generalization. We test this premise through controlled experiments that systematically vary data scale, concept diversity, and combination coverage. We find that compositional generalization is driven by data diversity, not mere data scale. Increased combinatorial coverage forces models to discover a linearly factored representational structure, where concepts decompose into additive components. We prove this structure is key to efficiency, enabling perfect generalization from few observed combinations. Evaluating pretrained models (DINO, CLIP), we find above-random yet imperfect performance, suggesting partial presence of this structure. Our work motivates stronger emphasis on constructing diverse datasets for compositional generalization, and considering the importance of representational structure that enables efficient compositional learning. Code available at https://github.com/oshapio/visual-compositional-generalization.

Citations

• On the rankability of visual embeddings
• Diffusion Classifiers Understand Compositionality, but Conditions Apply
• Intermediate Layer Classifiers for OOD generalization
• Pretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks
• CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally
• A Complexity-Based Theory of Compositionality
• Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics
• Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models
• Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks
• When does compositional structure yield compositional generalization? A kernel theory
• No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
• Compositional Generative Modeling: A Single Model is Not All You Need
• Data Factors for Better Compositional Generalization
• Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings
• Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models
• Provable Compositional Generalization for Object-Centric Learning
• PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning
• A Theory for Emergence of Complex Skills in Language Models
• Learning Conditional Attributes for Compositional Zero-Shot Learning
• DINOv2: Learning Robust Visual Features without Supervision
• Concept Algebra for (Score-Based) Text-Controlled Generative Models
• Break It Down: Evidence for Structural Compositionality in Neural Networks
• When and why vision-language models behave like bags-of-words, and what to do about it?
• Disentanglement with Biological Constraints: A Theory of Functional Cell Types
• A Benchmark for Compositional Visual Reasoning
• Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
• Training Compute-Optimal Large Language Models
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Emerging Properties in Self-Supervised Vision Transformers
• On the Transfer of Disentangled Representations in Realistic Settings
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• When and how CNNs generalize to out-of-distribution category-viewpoint combinations
• In Search of Lost Domain Generalization
• A causal view of compositional zero-shot recognition
• Language Models are Few-Shot Learners
• Shortcut learning in deep neural networks
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Disentangling by Factorising
• Deep Learning Scaling is Predictable, Empirically
• Teaching Compositionality to CNNs

Cited by

• Learning by Analogy: A Causal Framework for Composition Generalization
• An Alternative Trajectory for Generative AI
• SCAR: A Characterization Scheme for Multi-Modal Dataset
• On the rankability of visual embeddings
• Foveated Probes Recover Localized Binding Information in Vision Foundation Models
• Diffusion Classifiers Understand Compositionality, but Conditions Apply

Related

• Scaling can lead to compositional generalization
• Scalable Evaluation and Neural Models for Compositional Generalization
• Compositional Generalization from First Principles
• When does compositional structure yield compositional generalization? A kernel theory
• Successes and Limitations of Object-centric Models at Compositional Generalisation