2020/05/01 by Hu Hu, Hu, Hu, Rui Zhao +7 · 1 citation
Computer Science · Engineering · #Artificial intelligence #Artificial neural network #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #Computer science #Connectionism #Encoder #End-to-end principle #FOS: Computer and information sciences #FOS: Electrical engineering #Initialization #Language model #Leverage (statistics) #Music and Audio Processing #Natural Language Processing Techniques #Recurrent neural network #Speech Recognition and Synthesis #Speech recognition #Word error rate #cs.CL #eess.AS #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2005.00572
published in arXiv (Cornell University) (Cornell University) · Accepted by ICASSP 2020
arxiv created 2020/05/01 · openalex publication_date 2020/05/01 · arxiv updated 2020/05/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Recently, the recurrent neural network transducer (RNN-T) architecture has become an emerging trend in end-to-end automatic speech recognition research due to its advantages of being capable for online streaming speech recognition. However, RNN-T training is made difficult by the huge memory requirements, and complicated neural structure. A common solution to ease the RNN-T training is to employ connectionist temporal classification (CTC) model along with RNN language model (RNNLM) to initialize the RNN-T parameters. In this work, we conversely leverage external alignments to seed the RNN-T model. Two different pre-training solutions are explored, referred to as encoder pre-training, and whole-network pre-training respectively. Evaluated on Microsoft 65,000 hours anonymized production data with personally identifiable information removed, our proposed methods can obtain significant improvement. In particular, the encoder pre-training solution achieved a 10% and a 8% relative word error rate reduction when compared with random initialization and the widely used CTC+RNNLM initialization strategy, respectively. Our solutions also significantly reduce the RNN-T model latency from the baseline.