Spoken Language Understanding on Unseen Tasks With In-Context Learning

2025/05/12 by Agrawal, Neeraj, Ganapathy, Sriram · 1 citation
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2505.07731

Abstract

Spoken language understanding (SLU) tasks involve diverse skills that probe the information extraction, classification and/or generation capabilities of models. In this setting, task-specific training data may not always be available. While traditional task-specific SLU models are unable to cater to such requirements, the speech-text large language models (LLMs) offer a promising alternative with emergent abilities. However, out of-the-box, our evaluations indicate that the zero/few-shot performance of prominent open-source speech-text LLMs on SLU tasks are not up to the mark. In this paper, we introduce a novel approach to robust task-agnostic fine-tuning using randomized class labels. With this proposed fine-tuning, we illustrate that the performance of the speech-text LLMs on an unseen task is significantly improved over standard approaches. Critically, the proposed approach avoids the requirement of task-specific data annotations for enabling new tasks in speech-text LLMs.

Cited by

• Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech data

Related

• Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech data
• Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems
• UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions
• Speech Model Pre-training for End-to-End Spoken Language Understanding
• SpeechLLMs for Large-scale Contextualized Zero-shot Slot Filling