Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Data

2025/06/23 by Yun Tang, Tang, Yun, Eesung Kim +3
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Neural Networks and Applications #Sound (cs.SD) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.19159

openalex publication_date 2025/06/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A joint speech and text optimization method is proposed for hybrid transducer and attention-based encoder decoder (TAED) modeling to leverage large amounts of text corpus and enhance ASR accuracy. The joint TAED (J-TAED) is trained with both speech and text input modalities together, while it only takes speech data as input during inference. The trained model can unify the internal representations from different modalities, and be further extended to text-based domain adaptation. It can effectively alleviate data scarcity for mismatch domain tasks since no speech data is required. Our experiments show J-TAED successfully integrates speech and linguistic information into one model, and reduce the WER by 5.8 ~12.8% on the Librispeech dataset. The model is also evaluated on two out-of-domain datasets: one is finance and another is named entity focused. The text-based domain adaptation brings 15.3% and 17.8% WER reduction on those two datasets respectively.

Citations

• Effective internal language model training and fusion for factorized transducer model
• SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation
• Improved Factorized Neural Transducer Model For text-only Domain Adaptation
• t-SOT FNT: Streaming Multi-talker ASR with Text-only Domain Adaptation Capability
• Prompting Large Language Models with Speech Recognition Abilities
• Text-only Domain Adaptation using Unified Speech-Text Representation in Transducer
• Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks
• 4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders
• Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models
• CTC Alignments Improve Autoregressive Translation
• SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
• Pruned RNN-T for fast, memory-efficient ASR training
• Unified Speech-Text Pre-training for Speech Translation and Recognition
• MAESTRO: Matched Speech Text Representations through Modality Matching
• Integrating Text Inputs For Training and Adapting RNN Transducer ASR Models
• Recent Advances in End-to-End Automatic Speech Recognition
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task
• SPGISpeech: 5,000 hours of transcribed financial audio for fully\n formatted end-to-end speech recognition
• Intermediate Loss Regularization for CTC-based Speech Recognition
• A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss
• On the Relationship between Self-Attention and Convolutional Layers
• NeMo: a toolkit for building AI applications using Neural Modules
• Investigating Multilingual NMT Representations at Scale
• Two-Pass End-to-End Speech Recognition
• Learning pronunciation from a foreign language in speech synthesis networks
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
• Self-Attention with Relative Position Representations
• Attention Is All You Need
• Layer Normalization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence Transduction with Recurrent Neural Networks

Related

• Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation
• Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks
• USTED: Improving ASR with a Unified Speech and Text Encoder-Decoder
• Comparing the Benefit of Synthetic Training Data for Various Automatic\n Speech Recognition Architectures
• Investigating Methods to Improve Language Model Integration for\n Attention-based Encoder-Decoder ASR Models