In-Context Collapse in Vision-Language Models and How to Mitigate it?

2026/08/03 by Mohammad Rostami
Computer Science · #cs.CV #cs.AI

paper · pdf

arxiv created 2026/08/03 · arxiv updated 2026/08/05

Abstract

Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied. We show the opposite: as demonstrations accumulate, a subset of VLMs undergo an in-context collapse, a sharp, sometimes catastrophic accuracy drop spanning synthetic classification, natural-image classification, and VQA benchmarks, in some models falling below chance while outputs remain well-formed. Across an open VLM panel (0.5B--11B) and a frontier model (Claude Sonnet 4.5), the collapse is graded. Two capabilities turn out to be dissociable: robustness to accumulating demonstrations and the ability to learn a novel rule in context, their combinations yield three reproducible regimes. A parameter-matched lesion-and-rescue causally localizes the collapse to the vision-language integration pathway: an adapter on the connector and early/mid layers restores genuine learning (remap accuracy 0.39!→!0.91 at 16 shots), while an equal-capacity adapter on the late readout does not. We propose CircA, whose core is a one-time integration vaccine: trained once on one synthetic task, it transfers collapse-resistance to unseen task families (chance→0.71/0.60 on CIFAR/Fashion). The layers best for in-context integration are not the layers best for weight-based consolidation, the late readout achieves higher accuracy and less forgetting at fewer parameters. The collapse is an integration failure at the vision--language interface, correctable by a lightweight, transferable intervention.

Citations

• CAMA: Enhancing Multimodal In-Context Learning with Context-Aware Modulated Attention
• Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models
• What's in the Image? A Deep-Dive into the Vision of Vision Language Models
• SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning
• ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• LLaVA-OneVision: Easy Visual Task Transfer
• The Llama 3 Herd of Models
• Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
• Many-Shot In-Context Learning in Multimodal Foundation Models
• VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning
• Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• In-Context Learning Creates Task Vectors
• Function Vectors in Large Language Models
• Orthogonal Subspace Learning for Language Model Continual Learning
• Investigating the Catastrophic Forgetting in Multimodal Large Language Models
• Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP
• Link-Context Learning for Multimodal LLMs
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Otter: A Multi-Modal Model with In-Context Instruction Tuning
• What Makes Good Examples for Visual In-Context Learning?
• Images Speak in Images: A Generalist Painter for In-Context Visual Learning
• In-context Learning and Induction Heads
• CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks
• Cognitively Inspired Learning of Incremental Drifting Concepts
• Multimodal Few-Shot Learning with Frozen Language Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Language Models are Few-Shot Learners
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• LAMOL: LAnguage MOdeling for Lifelong Language Learning
• Efficient Lifelong Learning with A-GEM
• Gradient Episodic Memory for Continual Learning
• Continual Learning with Deep Generative Replay
• Overcoming catastrophic forgetting in neural networks
• iCaRL: Incremental Classifier and Representation Learning
• Learning without Forgetting