MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks
2025/07/31 by Niu, Yadong, Wang, Tianzi, Dinkel, Heinrich +7 · 2 citations
#Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering
paper · doi:10.48550/arxiv.2507.23511
Abstract
While large audio-language models have advanced open-ended audio understanding, they still fall short of nuanced human-level comprehension. This gap persists largely because current benchmarks, limited by data annotations and evaluation metrics, fail to reliably distinguish between generic and highly detailed model outputs. To this end, this work introduces MECAT, a Multi-Expert Constructed Benchmark for Fine-Grained Audio Understanding Tasks. Generated via a pipeline that integrates analysis from specialized expert models with Chain-of-Thought large language model reasoning, MECAT provides multi-perspective, fine-grained captions and open-set question-answering pairs. The benchmark is complemented by a novel metric: DATE (Discriminative-Enhanced Audio Text Evaluation). This metric penalizes generic terms and rewards detailed descriptions by combining single-sample semantic similarity with cross-sample discriminability. A comprehensive evaluation of state-of-the-art audio models is also presented, providing new insights into their current capabilities and limitations. The data and code are available at https://github.com/xiaomi-research/mecat
Citations
- Qwen3-Omni Technical Report
- Step-Audio 2 Technical Report
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- GLAP: General contrastive audio-text pretraining across domains and languages
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
- Kimi-Audio Technical Report
- SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning
- Qwen2.5-Omni Technical Report
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- Towards Unified Music Emotion Recognition across Dimensional and Categorical Models
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
- Baichuan-Omni Technical Report
- Sound-VECaps: Improving Audio Generation with Visual Enhanced Captions
- Open-Source Conversational AI with SpeechBrain 1.0
- AudioBench: A Universal Benchmark for Audio Large Language Models
- Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding
- Scaling up masked audio encoder learning for general audio classification
- FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model
- EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning
- emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
- The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation
- Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
- LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT
- Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning
- BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing
- LLaSM: Large Language and Speech Model
- CED: Consistent ensemble distillation for audio tagging
- LP-MusicCaps: LLM-Based Pseudo Music Captioning
- All-In-One Metrical And Functional Structure Analysis With Neighborhood Attentions on Demixed Audio
- Speech-based Age and Gender Prediction with Transformers
- AudioPaLM: A Large Language Model That Can Speak and Listen
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
- CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voice
- Pengi: An Audio Language Model for Audio Tasks
- X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
- AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
- Robust Speech Recognition via Large-Scale Weak Supervision
- Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering
- Learning to Answer Questions in Dynamic Audio-Visual Scenarios
- Can Audio Captions Be Evaluated with Image Caption Metrics?
- DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors
- SpeechBrain: A General-Purpose Speech Toolkit
- ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning
- DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to\n evaluate Noise Suppressors
- Clotho: An Audio Captioning Dataset
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- SPICE: Semantic Propositional Image Caption Evaluation
- CIDEr: Consensus-based Image Description Evaluation
Cited by
Related