The Evolved Transformer

2019/01/30 by David R. So, Chen Liang, Liang Chen +4 · 1 voice · 55 citations
Computer Science · Materials Science · Mathematics · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Magnetic Properties and Applications #Neural and Evolutionary Computing (cs.NE) #cs.CL #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1901.11117

ICML version with SOTA results

openalex publication_date 2019/01/30 · arxiv published 2019/01/30 · arxiv created 2019/05/17 · arxiv updated 2019/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models. Our goal is to apply NAS to search for a better alternative to the Transformer. We first construct a large search space inspired by the recent advances in feed-forward sequence models and then run evolutionary architecture search with warm starting by seeding our initial population with the Transformer. To directly search on the computationally expensive WMT 2014 English-German translation task, we develop the Progressive Dynamic Hurdles method, which allows us to dynamically allocate more resources to more promising candidate models. The architecture found in our experiments -- the Evolved Transformer -- demonstrates consistent improvement over the Transformer on four well-established language tasks: WMT 2014 English-German, WMT 2014 English-French, WMT 2014 English-Czech and LM1B. At a big model size, the Evolved Transformer establishes a new state-of-the-art BLEU score of 29.8 on WMT'14 English-German; at smaller sizes, it achieves the same quality as the original "big" Transformer with 37.6% less parameters and outperforms the Transformer by 0.7 BLEU at a mobile-friendly model size of 7M parameters.

Cited by

• Searching Meta Reasoning Skeleton to Guide LLM Reasoning
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
• A Large-Scale Chinese Short-Text Conversation Dataset
• A Survey on Neural Architecture Search
• Rethinking Skip Connection with Layer Normalization in Transformers and ResNets
• Towards Accurate and Compact Architectures via Neural Architecture Transformer
• Rapid Model Architecture Adaption for Meta-Learning
• Vocabulary Learning via Optimal Transport for Neural Machine Translation
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Scalable Transformers for Neural Machine Translation
• Taming Sparsely Activated Transformer with Stochastic Experts
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• Recurrent multiple shared layers in Depth for Neural Machine Translation
• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
• Towards Efficient Post-training Quantization of Pre-trained Language Models
• Towards Execution-Grounded Automated AI Research
• Incorporating BERT into Neural Machine Translation
• DeLighT: Deep and Light-weight Transformer
• The Benchmark Lottery
• Evolving Multi-Resolution Pooling CNN for Monaural Singing Voice Separation
• MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
• Learning Algebraic Recombination for Compositional Generalization
• ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
• Balanced One-shot Neural Architecture Optimization
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces
• Towards a Human-like Open-Domain Chatbot
• Residual Tree Aggregation of Layers for Neural Machine Translation
• Semi-Supervised Neural Architecture Search
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Learning Architectures from an Extended Search Space for Language Modeling
• Proof of Learning (PoLe): Empowering Machine Learning with Consensus Building on Blockchains
• Neural Machine Translation: A Review and Survey
• UdS Submission for the WMT 19 Automatic Post-Editing Task
• Selective Knowledge Distillation for Neural Machine Translation
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• AutoTrans: Automating Transformer Design via Reinforced Architecture Search
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• IOT: Instance-wise Layer Reordering for Transformer Structures
• Scaling Image and Video Generation via Test-Time Evolutionary Search
• AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer
• Extremely Low Footprint End-to-End ASR System for Smart Device
• Multi-branch Attentive Transformer
• Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations
• Neural Language Generation: Formulation, Methods, and Evaluation
• Evolving to learn: discovering interpretable plasticity rules for spiking networks
• RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models
• Evaluating Empathetic Chatbots in Customer Service Settings
• ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation
• Improved Conformer-based End-to-End Speech Recognition Using Neural Architecture Search
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding
• AutoRC: Improving BERT Based Relation Classification Models via Architecture Search
• Reinforced Curriculum Learning on Pre-trained Neural Machine Translation Models
• An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation
• Defending Deep Neural Networks against Backdoor Attacks via Module Switching

Discussions

• The Evolved Transformer [hn, 3 points, 0 comments]

Related

• Neural Architecture Search on Efficient Transformers and Beyond
• AutoTrans: Automating Transformer Design via Reinforced Architecture Search
• On Optimal Transformer Depth for Low-Resource Language Translation
• Very Deep Transformers for Neural Machine Translation
• Efficient Transformers: A Survey