Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection

2024/01/10 by Koji Inoue, Inoue, Koji, Bing’er Jiang +7 · 4 citations
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Human-Computer Interaction (cs.HC) #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2401.04868

openalex publication_date 2024/01/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialogue stereo audio to future voice activities. The VAP model includes contrastive predictive coding (CPC) and self-attention transformers, followed by a cross-attention transformer. We examine the effect of the input context audio length and demonstrate that the proposed system can operate in real-time with CPU settings, with minimal performance degradation.

Cited by

• Low-Latency Turn-Taking via Context-Aware Preface Generation in a Real-World Dialogue Robot
• Proactive Hearing Assistants that Isolate Egocentric Conversations
• Everything Counts: The Managed Omnirelevance of Speech in Human-Voice Agent Interaction
• Real-time Generation of Various Types of Nodding for Avatar Attentive Listening System

Related

• Voice Activity Projection: Self-supervised Learning of Turn-taking Events
• Prompt-Guided Turn-Taking Prediction
• Voice Activity Projection Model with Multimodal Encoders
• Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection
• Multilingual Turn-taking Prediction Using Voice Activity Projection