pyannote.audio: neural building blocks for speaker diarization

2019/11/04 by Bredin, Hervé, Yin, Ruiqing, Coria, Juan Manuel +7 · 26 citations
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.1911.01255

Abstract

We introduce pyannote.audio, an open-source toolkit written in Python for speaker diarization. Based on PyTorch machine learning framework, it provides a set of trainable end-to-end neural building blocks that can be combined and jointly optimized to build speaker diarization pipelines. pyannote.audio also comes with pre-trained models covering a wide range of domains for voice activity detection, speaker change detection, overlapped speech detection, and speaker embedding -- reaching state-of-the-art performance for most of them.

Cited by

• Towards Operational Conversational Intelligence: A Speech Intelligence Framework
• EmotionAI: A Privacy-Preserving Computational Intelligence Pipeline for Speech-Emotion-Grounded Conversational Analysis
• Chunking Strategies for Multimodal AI Systems
• Point of Order: Action-Aware LLM Persona Modeling for Data-Grounded Civic Deliberation
• MF-GCN: A Multi-Frequency Graph Convolutional Network for Tri-Modal Depression Detection Using Eye-Tracking, Facial, and Acoustic Features
• Emotion Recognition in Multi-Speaker Conversations through Speaker Identification, Knowledge Distillation, and Hierarchical Fusion
• VAD to the Bone: Ultra-Tiny Speech Activity Detection for Edge Deployment
• M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models
• Disentanglement of Sources in a Multi-Stream Variational Autoencoder
• Linguistic and Audio Embedding-Based Machine Learning for Alzheimer's Dementia and Mild Cognitive Impairment Detection: Insights from the PROCESS Challenge
• Target speaker anonymization in multi-speaker recordings
• Can Speech LLMs Think while Listening?
• Benchmarking Diarization Models
• DeepFake Detection in Dyadic Video Calls using Point of Gaze Tracking
• MMedFD: A Real-world Healthcare Benchmark for Multi-turn Full-Duplex Automatic Speech Recognition
• The MADRS Pipeline: Supporting Depression Assessment in Clinical Trials
• ClassMind: Scaling Classroom Observation and Instructional Feedback with Multimodal AI
• MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech
• Llama-Mimi: Speech Language Models with Interleaved Semantic and Acoustic Tokens
• ERFC: Happy Customers with Emotion Recognition and Forecasting in Conversation in Call Centers
• Beamforming-LLM: What, Where and When Did I Miss?
• Ensembling Synchronisation-based and Face-Voice Association Paradigms for Robust Active Speaker Detection in Egocentric Recordings
• Think Before You Talk: Enhancing Meaningful Dialogue Generation in Full-Duplex Speech Language Models with Planning-Inspired Text Guidance
• SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models
• Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization
• Assessment of Personality Dimensions Across Situations Using Conversational Speech

Related

• End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification
• Sequence-to-Sequence Neural Diarization with Automatic Speaker Detection and Representation
• A Review of Speaker Diarization: Recent Advances with Deep Learning
• SDBench: A Comprehensive Benchmark Suite for Speaker Diarization
• Speaker Diarization: Using Recurrent Neural Networks