Using multi-task learning to improve the performance of acoustic-to-word\n and conventional hybrid models

2019/02/02 by Thai‐Son Nguyen, Nguyen, Thai-Son, Sebastian Stueker +3
Computer Science · #Speech Recognition and Synthesis #Speech and Audio Processing #Music and Audio Processing

paper · pdf · doi:10.48550/arxiv.1902.01951

Abstract

Acoustic-to-word (A2W) models that allow direct mapping from acoustic signals\nto word sequences are an appealing approach to end-to-end automatic speech\nrecognition due to their simplicity. However, prior works have shown that\nmodelling A2W typically encounters issues of data sparsity that prevent\ntraining such a model directly. So far, pre-training initialization is the only\napproach proposed to deal with this issue. In this work, we propose to build a\nshared neural network and optimize A2W and conventional hybrid models in a\nmulti-task manner. Our results show that training an A2W model is much more\nstable with our multi-task model without pre-training initialization, and\nresults in a significant improvement compared to a baseline model. Experiments\nalso reveal that the performance of a hybrid acoustic model can be further\nimproved when jointly training with a sequence-level optimization criterion\nsuch as acoustic-to-word.\n

Related

• On Modular Training of Neural Acoustics-to-Word Model for LVCSR
• Toward Cross-Domain Speech Recognition with End-to-End Models
• Modular End-to-end Automatic Speech Recognition Framework for Acoustic-to-word Model
• Building competitive direct acoustics-to-word models for English\n conversational speech recognition
• Acoustic-to-Word Recognition with Sequence-to-Sequence Models