2025/02/12 by Junjie Wu, Wu, Junjie, Yumeng Fu +5 · 2 citations
Computer Science · Social Sciences · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Misinformation and Its Impacts #Multimedia (cs.MM) #Public Relations and Crisis Communication #Sentiment Analysis and Opinion Mining
paper · pdf · doi:10.48550/arxiv.2502.10455
openalex publication_date 2025/02/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent studies in Large Vision-Language Models (LVLMs) have demonstrated impressive advancements in multimodal Out-of-Context (OOC) misinformation detection, discerning whether an authentic image is wrongly used in a claim. Despite their success, the textual evidence of authentic images retrieved from the inverse search is directly transmitted to LVLMs, leading to inaccurate or false information in the decision-making phase. To this end, we present E2LVLM, a novel evidence-enhanced large vision-language model by adapting textual evidence in two levels. First, motivated by the fact that textual evidence provided by external tools struggles to align with LVLMs inputs, we devise a reranking and rewriting strategy for generating coherent and contextually attuned content, thereby driving the aligned and effective behavior of LVLMs pertinent to authentic images. Second, to address the scarcity of news domain datasets with both judgment and explanation, we generate a novel OOC multimodal instruction-following dataset by prompting LVLMs with informative content to acquire plausible explanations. Further, we develop a multimodal instruction-tuning strategy with convincing explanations for beyond detection. This scheme contributes to E2LVLM for multimodal OOC misinformation detection and explanation. A multitude of experiments demonstrate that E2LVLM achieves superior performance than state-of-the-art methods, and also provides compelling rationales for judgments.