On decoder-only architecture for speech-to-text and large language model integration

2023/07/08 by Jian Wu, Yashesh Gaur, Wu, Jian +19 · 31 citations
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #Topic Modeling #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2307.03917

openalex publication_date 2023/07/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have achieved remarkable success in the field of natural language processing, enabling better human-computer interaction using natural language. However, the seamless integration of speech signals into LLMs has not been explored well. The "decoder-only" architecture has also not been well studied for speech processing tasks. In this research, we introduce Speech-LLaMA, a novel approach that effectively incorporates acoustic information into text-based large language models. Our method leverages Connectionist Temporal Classification and a simple audio encoder to map the compressed acoustic features to the continuous semantic space of the LLM. In addition, we further probe the decoder-only architecture for speech-to-text tasks by training a smaller scale randomly initialized speech-LLaMA model from speech-text paired data alone. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement over strong baselines, highlighting the potential advantages of decoder-only models for speech-to-text conversion.

Cited by

• Do modern speech LLMs and re-scoring techniques improve bilingual ASR performance for Basque and Spanish in domain-specific contexts?
• Adaptive Edge-Cloud Inference for Speech-to-Action Systems Using ASR and Large Language Models
• Protecting Bystander Privacy via Selective Hearing in Audio LLMs
• ZO-ASR: Zeroth-Order Fine-Tuning of Speech Foundation Models without Back-Propagation
• Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation
• Towards Effective and Efficient Non-autoregressive decoders for Conformer and LLM-based ASR using Block-based Attention Mask
• Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking
• Abjad AI at NADI 2025: CATT-Whisper: Multimodal Diacritic Restoration Using Text and Speech Representations
• PFEA: An LLM-based High-Level Natural Language Planning and Feedback Embodied Agent for Human-Centered AI
• Slot Filling as a Reasoning Task for SpeechLLMs
• SpeechLLMs for Large-scale Contextualized Zero-shot Slot Filling
• StressTransfer: Stress-Aware Speech-to-Speech Translation with Emphasis Preservation
• End-to-end Automatic Speech Recognition and Speech Translation: Integration of Speech Foundational Models and LLMs
• MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
• Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?
• When Voice Matters: Evidence of Gender Disparity in Positional Bias of SpeechLLMs
• Enhancing Speech Large Language Models through Reinforced Behavior Alignment
• Speech Language Models for Under-Represented Languages: Insights from Wolof
• ENJ: Optimizing Noise with Genetic Algorithms to Jailbreak LSMs
• FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations
• Denoising GER: A Noise-Robust Generative Error Correction with LLM for Speech Recognition
• Group Relative Policy Optimization for Speech Recognition
• Optimal Transport Regularization for Speech Text Alignment in Spoken Language Models
• Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages
• SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding
• The TEA-ASLP System for Multilingual Conversational Speech Recognition and Speech Diarization in MLC-SLM 2025 Challenge
• video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models
• Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition
• Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs
• Bridging the Modality Gap: Softly Discretizing Audio Representation for LLM-based Automatic Speech Recognition
• LLM-based phoneme-to-grapheme for phoneme-based speech recognition

Related

• Investigating Decoder-only Large Language Models for Speech-to-text Translation
• Large Language Model Guided Decoding for Self-Supervised Speech Recognition
• Prompting Large Language Models with Speech Recognition Abilities
• AudioPaLM: A Large Language Model That Can Speak and Listen
• OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification