MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization

2025/10/02 by Liu, Yinhong, He, Jianfeng, Su, Hang +6 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.01659

Abstract

Multimodal Dialogue Summarization (MDS) is a critical task with wide-ranging applications. To support the development of effective MDS models, robust automatic evaluation methods are essential for reducing both cost and human effort. However, such methods require a strong meta-evaluation benchmark grounded in human annotations. In this work, we introduce MDSEval, the first meta-evaluation benchmark for MDS, consisting image-sharing dialogues, corresponding summaries, and human judgments across eight well-defined quality aspects. To ensure data quality and richfulness, we propose a novel filtering framework leveraging Mutually Exclusive Key Information (MEKI) across modalities. Our work is the first to identify and formalize key evaluation dimensions specific to MDS. We benchmark state-of-the-art modal evaluation methods, revealing their limitations in distinguishing summaries from advanced MLLMs and their susceptibility to various bias.

Citations

• Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
• Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets
• TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning
• MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
• Large Language Models Are Not Robust Multiple Choice Selectors
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Large Language Models are not Fair Evaluators
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos
• Large Scale Multi-Lingual Multi-Modal Summarization Dataset
• A Survey on In-context Learning
• DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset
• MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation
• Towards a Unified Multi-Dimensional Evaluator for Text Generation
• Constructing Multi-Modal Dialogue Dataset by Replacing Text with\n Semantically Relevant Images
• PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling
• Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary
• SummEval: Re-evaluating Summarization Evaluation
• USR: An Unsupervised and Reference Free Evaluation Metric for Dialog\n Generation
• VD-BERT: A Unified Vision and Dialog Transformer with BERT
• Multimodal Logical Inference System for Visual-Textual Entailment
• Visual Coreference Resolution in Visual Dialog using Neural Module Networks
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• DialogSum: A Real-Life Scenario Dialogue Summarization Dataset

Cited by

• MMLDSum-LLM: Multimodal Long-Document Summarization with Visual-Alignment and Keyword-Aware

Related

• Two eyes, Two views, and finally, One summary! Towards Multi-modal Multi-tasking Knowledge-Infused Medical Dialogue Summarization
• DialogSum Challenge: Results of the Dialogue Summarization Shared Task
• LLMs as Architects and Critics for Multi-Source Opinion Summarization
• Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization
• A Survey on Multi-modal Summarization