Preservation of Language Understanding Capabilities in Speech-aware Large Language Models

2025/09/15 by Marek Kubis, Kubis, Marek, Paweł Skórzewski +11
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2509.12171

openalex publication_date 2025/09/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The paper presents C3T (Cross-modal Capabilities Conservation Test), a new benchmark for assessing the performance of speech-aware large language models. The benchmark utilizes textual tasks and a voice cloning text-to-speech model to quantify the extent to which language understanding capabilities are preserved when the model is accessed via speech input. C3T quantifies the fairness of the model for different categories of speakers and its robustness across text and speech modalities.

Citations

• Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
• The Llama 3 Herd of Models
• AI Benchmark Half-Life in Recursive Corpora: A Theory of Validity Decay under Semantic Leakage and Regeneration
• LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
• AudioBench: A Universal Benchmark for Audio Large Language Models
• GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech
• LLM Evaluators Recognize and Favor Their Own Generations
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
• Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
• Evaluating large language models in theory of mind tasks
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Flow Matching for Generative Modeling
• Training Verifiers to Solve Math Word Problems
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Measuring Mathematical Problem Solving With the MATH Dataset
• Measuring Massive Multitask Language Understanding
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Attention Is All You Need
• Counterfactual Fairness

Related

• Cross-Modal Knowledge Distillation for Speech Large Language Models
• Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond
• Roadmap towards Superhuman Speech Understanding using Large Language Models
• Proceedings of the ISCA/ITG Workshop on Diversity in Large Speech and Language Models
• Closing the Gap Between Text and Speech Understanding in LLMs