Rethinking Perturbations in Encoder-Decoders for Fast Training

2021/04/05 by Sho Takase, Takase, Sho, Shun Kiyono +1 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2104.01853

openalex publication_date 2021/04/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbations (Sato et al., 2019) as perturbations but these methods require considerable computational time. Thus, this study addresses the question of whether these approaches are efficient enough for training time. We compare several perturbations in sequence-to-sequence problems with respect to computational time. Experimental results show that the simple techniques such as word dropout (Gal and Ghahramani, 2016) and random replacement of input tokens achieve comparable (or better) scores to the recently proposed perturbations, even though these simple methods are faster. Our code is publicly available at https://github.com/takase/rethink_perturbations.

Citations

• Sequence to Sequence Learning with Neural Networks
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Parallel Scheduled Sampling

Cited by

• Self-supervised Latent Space Optimization with Nebula Variational Coding

Related

• Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
• Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training
• Dynamic Scheduled Sampling with Imitation Loss for Neural Text Generation
• ARCH: Efficient Adversarial Regularized Training with Caching
• Online Segment to Segment Neural Transduction