Glancing Transformer for Non-Autoregressive Neural Machine Translation

2020/08/18 by Lihua Qian, Qian, Lihua, Hao Zhou +13 · 10 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2008.07905

9 pages, 7 figures, ACL2021

openalex publication_date 2020/08/18 · arxiv created 2021/05/13 · arxiv updated 2021/05/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent work on non-autoregressive neural machine translation (NAT) aims at improving the efficiency by parallel decoding without sacrificing the quality. However, existing NAT methods are either inferior to Transformer or require multiple decoding passes, leading to reduced speedup. We propose the Glancing Language Model (GLM), a method to learn word interdependency for single-pass parallel generation models. With GLM, we develop Glancing Transformer (GLAT) for machine translation. With only single-pass parallel decoding, GLAT is able to generate high-quality translation with 8-15 times speedup. Experiments on multiple WMT language directions show that GLAT outperforms all previous single pass non-autoregressive methods, and is nearly comparable to Transformer, reducing the gap to 0.25-0.9 BLEU points.

Citations

• Adam: A Method for Stochastic Optimization
• Non-Autoregressive Neural Machine Translation
• Imitation Learning for Non-Autoregressive Neural Machine Translation
• Non-autoregressive Transformer by Position Learning
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation
• Aligned Cross Entropy for Non-Autoregressive Machine Translation
• Cascaded Text Generation with Markov Transformers
• An EM Approach to Non-autoregressive Conditional Sequence Generation

Cited by

• Towards Effective and Efficient Non-autoregressive decoders for Conformer and LLM-based ASR using Block-based Attention Mask
• Bidirectional Representations Augmented Autoregressive Biological Sequence Generation
• Transformer-Encoder Trees for Efficient Multilingual Machine Translation and Speech Translation
• Enhancing the Robustness of Contextual ASR to Varying Biasing Information Volumes Through Purified Semantic Correlation Joint Modeling
• Confidence-Modulated Speculative Decoding for Large Language Models
• Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
• Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
• Pseudo Labels-based Neural Speech Enhancement for the AVSR Task in the MISP-Meeting Challenge
• Improving Non-autoregressive Generation with Mixup Training
• DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Related

• Non-Autoregressive Neural Machine Translation
• Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation
• Semi-Autoregressive Neural Machine Translation
• Weighted Transformer Network for Machine Translation
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation