Multi-scale Alignment and Contextual History for Attention Mechanism in Sequence-to-sequence Model

2018/07/22 by Tjandra, Andros, Sakti, Sakriani, Nakamura, Satoshi
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.1807.08280

Abstract

A sequence-to-sequence model is a neural network module for mapping two sequences of different lengths. The sequence-to-sequence model has three core modules: encoder, decoder, and attention. Attention is the bridge that connects the encoder and decoder modules and improves model performance in many tasks. In this paper, we propose two ideas to improve sequence-to-sequence model performance by enhancing the attention module. First, we maintain the history of the location and the expected context from several previous time-steps. Second, we apply multiscale convolution from several previous attention vectors to the current decoder state. We utilized our proposed framework for sequence-to-sequence speech recognition and text-to-speech systems. The results reveal that our proposed extension could improve performance significantly compared to a standard attention baseline.

Related

• Supervised Attention in Sequence-to-Sequence Models for Speech Recognition
• Attention Strategies for Multi-Source Sequence-to-Sequence Learning
• Temporal Convolutional Attention-based Network For Sequence Modeling
• On using 2D sequence-to-sequence models for speech recognition
• Neural Attention Models for Sequence Classification: Analysis and Application to Key Term Extraction and Dialogue Act Detection