SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models

2025/08/08 by Yin Han, Yin, Han, Yafeng Chen +15 · 5 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Joint (building) #Language model #Mechanism (biology) #Natural Language Processing Techniques #Sound (cs.SD) #Speaker diarisation #Speaker recognition #Speech Recognition and Synthesis #Task (project management) #Task analysis #Transcription (linguistics)

paper · pdf · doi:10.48550/arxiv.2508.06372

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/08/08 · openalex created_date 2025/10/15 · openalex updated_date 2026/08/05

Abstract

The Speaker Diarization and Recognition (SDR) task aims to predict "who spoke when and what" within an audio clip, which is a crucial task in various real-world multi-speaker scenarios such as meeting transcription and dialogue systems. Existing SDR systems typically adopt a cascaded framework, combining multiple modules such as speaker diarization (SD) and automatic speech recognition (ASR). The cascaded systems suffer from several limitations, such as error propagation, difficulty in handling overlapping speech, and lack of joint optimization for exploring the synergy between SD and ASR tasks. To address these limitations, we introduce SpeakerLM, a unified multimodal large language model for SDR that jointly performs SD and ASR in an end-to-end manner. Moreover, to facilitate diverse real-world scenarios, we incorporate a flexible speaker registration mechanism into SpeakerLM, enabling SDR under different speaker registration settings. SpeakerLM is progressively developed with a multi-stage training strategy on large-scale real data. Extensive experiments show that SpeakerLM demonstrates strong data scaling capability and generalizability, outperforming state-of-the-art cascaded baselines on both in-domain and out-of-domain public SDR benchmarks. Furthermore, experimental results show that the proposed speaker registration mechanism effectively ensures robust SDR performance of SpeakerLM across diverse speaker registration conditions and varying numbers of registered speakers.

Citations

• Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization
• AISHELL-5: The First Open-Source In-Car Multi-Channel Multi-Speaker Speech Dataset for Automatic Speech Diarization and Recognition
• The Multimodal Information Based Speech Processing (MISP) 2025 Challenge: Audio-Visual Diarization and Recognition
• Kimi-Audio Technical Report
• MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
• Leveraging Self-Supervised Learning for Speaker Diarization
• Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions
• ERes2NetV2: Boosting Short-Duration Speaker Verification Performance with Computational Efficiency
• 3D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization
• Towards Understanding Convergence and Generalization of AdamW
• The second multi-channel multi-party meeting transcription challenge (M2MeT) 2.0): A benchmark for speaker-attributed ASR
• Enhancing Speaker Diarization with Large Language Models: A Contextual Beam Search Approach
• The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios
• CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking
• From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization
• Summary On The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Grand Challenge
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech
• AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario
• A Review of Speaker Diarization: Recent Advances with Deep Learning
• Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis
• pyannote.audio: neural building blocks for speaker diarization
• End-to-End Neural Speaker Diarization with Self-attention
• AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale
• Attention Is All You Need

Cited by

• Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio
• Benchmarking Humans and Machines on Complex Multilingual Speech Understanding Tasks
• From Who Said What to Who They Are: Modular Training-free Identity-Aware LLM Refinement of Speaker Diarization
• Can Large Audio Language Models Understand Audio Well? Speech, Scene and Events Understanding Benchmark for LALMs
• VIBEVOICE-ASR Technical Report

Related

• One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognition
• Diarization-Aware Multi-Speaker Automatic Speech Recognition via Large Language Models
• Speech Recognition and Multi-Speaker Diarization of Long Conversations
• A Review of Speaker Diarization: Recent Advances with Deep Learning
• DiCoW: Diarization-Conditioned Whisper for Target Speaker Automatic Speech Recognition