2021/04/02 by Qingyun Dou, Dou, Qingyun, Yiting Lu +8 · 2 citations
Chemistry · Computer Science · #Artificial Intelligence (cs.AI) #Chemistry #Climatology #Computation and Language (cs.CL) #Computer science #FOS: Computer and information sciences #Forcing (mathematics) #Geology #History #Machine Learning (cs.LG) #Machine translation #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language processing #Topic Modeling #Translation (biology) #cs.AI #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2104.01264
published in arXiv (Cornell University) (Cornell University) · arXiv admin note: text overlap with arXiv:1909.12289
arxiv created 2021/04/02 · openalex publication_date 2021/04/02 · arxiv updated 2021/04/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Auto-regressive sequence-to-sequence models with attention mechanisms have achieved state-of-the-art performance in various tasks including Text-To-Speech (TTS) and Neural Machine Translation (NMT). The standard training approach, teacher forcing, guides a model with the reference output history. At inference stage, the generated output history must be used. This mismatch can impact performance. However, it is highly challenging to train the model using the generated output. Several approaches have been proposed to address this problem, normally by selectively using the generated output history. To make training stable, these approaches often require a heuristic schedule or an auxiliary classifier. This paper introduces attention forcing for NMT. This approach guides the model with the generated output history and reference attention, and can reduce the training-inference mismatch without a schedule or a classifier. Attention forcing has been successful in TTS, but its application to NMT is more challenging, due to the discrete and multi-modal nature of the output space. To tackle this problem, this paper adds a selection scheme to vanilla attention forcing, which automatically selects a suitable training approach for each pair of training data. Experiments show that attention forcing can improve the overall translation quality and the diversity of the translations.