DeLighT: Deep and Light-weight Transformer

2020/08/03 by Sachin Mehta, Marjan Ghazvininejad, Mehta, Sachin +7 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2008.00623

openalex publication_date 2020/08/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a deep and light-weight transformer, DeLighT, that delivers similar or better performance than standard transformer-based models with significantly fewer parameters. DeLighT more efficiently allocates parameters both (1) within each Transformer block using the DeLighT transformation, a deep and light-weight transformation, and (2) across blocks using block-wise scaling, which allows for shallower and narrower DeLighT blocks near the input and wider and deeper DeLighT blocks near the output. Overall, DeLighT networks are 2.5 to 4 times deeper than standard transformer models and yet have fewer parameters and operations. Experiments on benchmark machine translation and language modeling tasks show that DeLighT matches or improves the performance of baseline Transformers with 2 to 3 times fewer parameters on average. Our source code is available at: \urlhttps://github.com/sacmehta/delight

Citations

• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• Benefits of depth in neural networks
• An Analysis of Neural Language Modeling at Multiple Scales
• Pay Less Attention with Lightweight and Dynamic Convolutions
• The Evolved Transformer
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• LLM-Supported Manufacturing Mapping Generation
• Reformer: The Efficient Transformer
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Longformer: The Long-Document Transformer
• Synthesizer: Rethinking Self-Attention in Transformer Models

Cited by

• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
• The Efficiency Misnomer

Related

• Transformer on a Diet
• OPT: Open Pre-trained Transformer Language Models
• TorchScale: Transformers at Scale
• Efficient Transformers: A Survey
• Transformer models: an introduction and catalog