2019/10/04 by Zhen Huang, Huang, Zhen, Tim Ng +10
Computer Science · Engineering · Mathematics · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #cs.CL #cs.LG #cs.SD #eess.AS #electronic engineering #information engineering #stat.ML
paper · pdf · doi:10.48550/arxiv.1910.01992
openalex publication_date 2019/10/04 · openalex created_date 2019/10/10 · arxiv created 2020/03/23 · arxiv updated 2020/03/25 · openalex updated_date 2026/07/28
Very deep CNNs achieve state-of-the-art results in both computer vision and speech recognition, but are difficult to train. The most popular way to train very deep CNNs is to use shortcut connections (SC) together with batch normalization (BN). Inspired by Self- Normalizing Neural Networks, we propose the self-normalizing deep CNN (SNDCNN) based acoustic model topology, by removing the SC/BN and replacing the typical RELU activations with scaled exponential linear unit (SELU) in ResNet-50. SELU activations make the network self-normalizing and remove the need for both shortcut connections and batch normalization. Compared to ResNet- 50, we can achieve the same or lower (up to 4.5% relative) word error rate (WER) while boosting both training and inference speed by 60%-80%. We also explore other model inference optimization schemes to further reduce latency for production use.