Safe Semantics, Unsafe Interpretations: Tackling Implicit Reasoning Safety in Large Vision-Language Models

2025/08/12 by Wei Cai, Cai, Wei, Jian Zhao +6
Computer Science · Social Sciences · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2508.08926

openalex publication_date 2025/08/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Vision-Language Models face growing safety challenges with multimodal inputs. This paper introduces the concept of Implicit Reasoning Safety, a vulnerability in LVLMs. Benign combined inputs trigger unsafe LVLM outputs due to flawed or hidden reasoning. To showcase this, we developed Safe Semantics, Unsafe Interpretations, the first dataset for this critical issue. Our demonstrations show that even simple In-Context Learning with SSUI significantly mitigates these implicit multimodal threats, underscoring the urgent need to improve cross-modal implicit reasoning.

Citations

• Mimic In-Context Learning for Multimodal Tasks
• Qwen2.5-VL Technical Report
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback
• Qwen Technical Report
• Egoshots, an ego-vision life-logging dataset and semantic fidelity\n metric to evaluate diversity in image captioning models
• Microsoft COCO: Common Objects in Context

Related

• When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models
• Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models
• Self-Aware Safety Augmentation: Leveraging Internal Semantic Understanding to Enhance Safety in Vision-Language Models
• Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models
• VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety