Distilling an End-to-End Voice Assistant Without Instruction Training Data

2024/10/03 by William A. Held, E Li, Held, William +9 · 7 citations
Computer Science · #AI in Service Interactions #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning

paper · pdf · doi:10.48550/arxiv.2410.02678

openalex publication_date 2024/10/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Voice assistants, such as Siri and Google Assistant, typically model audio and text separately, resulting in lost speech information and increased complexity. Recent efforts to address this with end-to-end Speech Large Language Models (LLMs) trained with supervised finetuning (SFT) have led to models ``forgetting" capabilities from text-only LLMs. Our work proposes an alternative paradigm for training Speech LLMs without instruction data, using the response of a text-only LLM to transcripts as self-supervision. Importantly, this process can be performed without annotated responses. We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation. Furthermore, we show that DiVA better meets user preferences, achieving a 72% win rate compared with state-of-the-art models like Qwen 2 Audio, despite using >100x less training compute.

Cited by

• SeaLLMs-Audio: Large Audio-Language Models for Southeast Asia
• SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
• Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking
• SARSteer: Safeguarding Large Audio-Language Models via Safe-Ablated Refusal Steering
• Closing the Gap Between Text and Speech Understanding in LLMs
• ENJ: Optimizing Noise with Genetic Algorithms to Jailbreak LSMs
• DIFFA: Large Language Diffusion Models Can Listen and Understand

Related

• Speechless: Speech Instruction Training Without Speech for Low Resource Languages
• Exploring Transfer Learning For End-to-End Spoken Language Understanding
• VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning
• Self-Improvement for Audio Large Language Model using Unlabeled Speech
• TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment