QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization

2021/04/13 by Ming Zhong, Da Yin, Zhong, Ming +20 · 67 citations
Computer Science · #Advanced Graph Neural Networks #Automatic summarization #Benchmark (surveying) #Computation and Language (cs.CL) #Computer science #Data science #Domain (mathematical analysis) #FOS: Computer and information sciences #Information retrieval #Key (lock) #Multi-document summarization #Natural Language Processing Techniques #Set (abstract data type) #Task (project management) #Topic Modeling #World Wide Web #cs.CL

paper · pdf · doi:10.48550/arxiv.2104.05938

published in arXiv (Cornell University) (Cornell University) · Accepted by NAACL 2021

arxiv created 2021/04/13 · openalex created_date 2021/04/13 · openalex publication_date 2021/04/13 · arxiv updated 2021/04/14 · openalex updated_date 2026/08/06

Abstract

Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard to create a single short summary that covers all the content of a long meeting involving multiple people and topics. In order to satisfy the needs of different types of users, we define a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and we introduce QMSum, a new benchmark for this task. QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple domains. Besides, we investigate a locate-then-summarize method and evaluate a set of strong summarization baselines on the task. Experimental results and manual analysis reveal that QMSum presents significant challenges in long meeting summarization for future research. Dataset is available at \urlhttps://github.com/Yale-LILY/QMSum.

Cited by

• Coordinated Networking for On-Device Agent-Augmented Real-Time Communication
• Enhancing Long Document Long Form Summarisation with Self-Planning
• Short-Context Dominance: How Much Local Context Natural Language Actually Needs?
• SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification
• WaterSearch: A Quality-Aware Search-based Watermarking Framework for Large Language Models
• Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings
• KV Cache Transform Coding for Compact Storage in LLM Inference
• Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
• Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
• APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training
• Evaluating Long-Term Memory for Long-Context Question Answering
• No Mean Feat: Simple, Strong Baselines for Context Compression
• Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
• Taming the Fragility of KV Cache Eviction in LLM Inference
• Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors
• DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations
• DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarization
• CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension
• Adaptive Planning for Multi-Attribute Controllable Summarization with Monte Carlo Tree Search
• SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
• CHOIR: A Chatbot-mediated Organizational Memory Leveraging Communication in University Research Labs
• When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models
• CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density
• Value-Guided KV Compression for LLMs via Approximated CUR Decomposition
• From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
• Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization
• HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models
• On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
• MSRS: Evaluating Multi-Source Retrieval-Augmented Generation
• AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
• SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation
• TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain
• AttnTrace: Attention-based Context Traceback for Long-Context LLMs
• Teaching Language Models To Gather Information Proactively
• From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation
• TalkLess: Blending Extractive and Abstractive Speech Summarization for Editing Speech to Preserve Content and Style
• Flora: Effortless Context Construction to Arbitrary Length and Scale
• Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks
• Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges
• StreamHover: Livestream Transcript Summarization and Annotation
• EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes
• SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression
• RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
• MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
• Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarization
• GenerationPrograms: Fine-grained Attribution with Executable Programs
• HeavyWater and SimplexWater: Distortion-Free LLM Watermarks for Low-Entropy Next-Token Predictions
• Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
• TODSum: Task-Oriented Dialogue Summarization with State Tracking
• Low-Resource Dialogue Summarization with Domain-Agnostic Multi-Source Pretraining
• TracLLM: A Generic Framework for Attributing Long Context LLMs
• HiCaM: A Hierarchical-Causal Modification Framework for Long-Form Text Modification
• From Chat Logs to Collective Insights: Aggregative Question Answering
• Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaboration
• MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models
• Multi-Party Conversational Agents: A Survey
• MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing
• An Exploratory Study on Long Dialogue Summarization: What Works and What's Next
• TIDES: A Longitudinal Bilingual Dataset for Modeling Multi-Party Social Dynamics
• Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answering
• Averaging Bias: Human Faithfulness Annotations are not Locally Faithful
• A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization
• Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM
• A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning
• TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
• Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuning
• Summarizing Speech: A Comprehensive Survey

Related

• Investigating Consistency in Query-Based Meeting Summarization: A Comparative Study of Different Embedding Methods
• Meeting Summarization with Pre-training and Clustering Methods
• Meeting Summarization: A Survey of the State of the Art
• Improving Query-Focused Meeting Summarization with Query-Relevant Knowledge
• Learning to Rank Utterances for Query-Focused Meeting Summarization