LLM-Synth4KWS: Scalable Automatic Generation and Synthesis of Confusable Data for Custom Keyword Spotting

2025/05/29 by Pai Zhu, Quan Wang, Zhu, Pai +5
Computer Science · Medicine · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #Voice and Speech Disorders #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2505.22995

openalex publication_date 2025/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Custom keyword spotting (KWS) allows detecting user-defined spoken keywords from streaming audio. This is achieved by comparing the embeddings from voice enrollments and input audio. State-of-the-art custom KWS models are typically trained contrastively using utterances whose keywords are randomly sampled from training dataset. These KWS models often struggle with confusing keywords, such as "blue" versus "glue". This paper introduces an effective way to augment the training with confusable utterances where keywords are generated and grouped from large language models (LLMs), and speech signals are synthesized with diverse speaking styles from text-to-speech (TTS) engines. To better measure user experience on confusable KWS, we define a new northstar metric using the average area under DET curve from confusable groups (c-AUC). Featuring high scalability and zero labor cost, the proposed method improves AUC by 3.7% and c-AUC by 11.3% on the Speech Commands testing set.

Citations

• GraphemeAug: A Systematic Approach to Synthesized Hard Negative Keyword Spotting Examples
• Phoneme-Level Contrastive Learning for User-Defined Keyword Spotting with Flexible Enrollment
• GE2E-KWS: Generalized End-to-End Training and Evaluation for Zero-shot Keyword Spotting
• Adversarial training of Keyword Spotting to Minimize TTS Data Overfitting
• Utilizing TTS Synthesized Data for Efficient Development of Keyword Spotting Model
• Synth4Kws: Synthesized Speech for User Defined Keyword Spotting in Low Resource Environments
• Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Personalizing Keyword Spotting with Speaker Information
• Locale Encoding For Scalable Multilingual Keyword Spotting Models
• Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech
• Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting
• MAESTRO: Matched Speech Text Representations through Modality Matching
• Generating Adversarial Samples For Training Wake-up Word Detection Systems Against Confusing Words
• Synth2Aug: Cross-domain speaker recognition with TTS synthesized speech
• Training Wake Word Detection with Synthesized Speech Data on Confusion Words
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Training Keyword Spotters with Limited and Synthesized Speech Data
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• End-to-End Streaming Keyword Spotting
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Generalized End-to-End Loss for Speaker Verification

Cited by

• GraphemeAug: A Systematic Approach to Synthesized Hard Negative Keyword Spotting Examples

Related

• Synth4Kws: Synthesized Speech for User Defined Keyword Spotting in Low Resource Environments
• Utilizing TTS Synthesized Data for Efficient Development of Keyword Spotting Model
• Deep Spoken Keyword Spotting: An Overview
• Contrastive Augmentation: An Unsupervised Learning Approach for Keyword Spotting in Speech Technology
• Keyword Spotting with Hyper-Matched Filters for Small Footprint Devices