MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation

2021/08/19 by Pan Xie, Xie, Pan, Zexian Li +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2108.08447

openalex publication_date 2021/08/19 · openalex created_date 2021/08/30 · openalex updated_date 2026/07/28

Abstract

Conditional masked language models (CMLM) have shown impressive progress in non-autoregressive machine translation (NAT). They learn the conditional translation model by predicting the random masked subset in the target sentence. Based on the CMLM framework, we introduce Multi-view Subset Regularization (MvSR), a novel regularization method to improve the performance of the NAT model. Specifically, MvSR consists of two parts: (1) shared mask consistency: we forward the same target with different mask strategies, and encourage the predictions of shared mask positions to be consistent with each other. (2) model consistency, we maintain an exponential moving average of the model weights, and enforce the predictions to be consistent between the average model and the online model. Without changing the CMLM-based architecture, our approach achieves remarkable performance on three public benchmarks with 0.36-1.14 BLEU gains over previous NAT models. Moreover, compared with the stronger Transformer baseline, we reduce the gap to 0.01-0.44 BLEU scores on small datasets (WMT16 RO↔EN and IWSLT DE→EN).

Citations

• Sequence to Sequence Learning with Neural Networks
• Convolutional Sequence to Sequence Learning
• Fast Decoding in Sequence Models using Discrete Latent Variables
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Cross-lingual Language Model Pretraining
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Imitation Learning for Non-Autoregressive Neural Machine Translation
• Adversarial Training for Large Neural Language Models
• Language Models are Few-Shot Learners
• Contrastive Multiview Coding
• R-Drop: Regularized Dropout for Neural Networks

Related

• Improving Non-autoregressive Machine Translation with Error Exposure and Consistency Regularization
• Incorporating a Local Translation Mechanism into Non-autoregressive Translation
• Non-Autoregressive Machine Translation with Auxiliary Regularization
• Mask-Predict: Parallel Decoding of Conditional Masked Language Models
• Inference Strategies for Machine Translation with Conditional Masking