Visual Saliency Steering Distillation for Multimodal Chain-of-Thought Reasoning

2026/04/21 by Hao Yang, Jin Wang, Yi-Xiang Wang +2
Computer Science · #Visual Attention and Saliency Detection #Multimodal Machine Learning Applications #Constraint Satisfaction and Optimization

paper · pdf · doi:10.1109/icassp55912.2026.11463006

Abstract

Multimodal chain-of-thought (CoT) reasoning integrates visual and textual cues through step-by-step inference. In small models with limited token budgets, modality-interaction fusion often suppresses tiny cross-modal differences. In particular, multimodal CoT often struggles when different images pair with identical text or different texts pair with an identical image, making such inputs nearly indistinguishable after fusion. This study proposes Visual Saliency Steering Distillation (VSSD). VSSD leverages the attention maps of multimodal large language models to generate perturbed images that capture task-sensitive feature directions, and then applies singular value decomposition to extract dominant steering vectors to guide inter-layer distillation. Experiments on ScienceQA and M<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup>CoT demonstrate that VSSD improves rationale generation and answer inference. The code is available at https://github.com/BGWH123/VSSD.

Citations

• Multimodal Chain-of-Thought Reasoning in Language Models

Related

• Interleaved-Modal Chain-of-Thought
• Multimodal Chain-of-Thought Reasoning in Language Models
• Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
• Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Space
• CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models