Depth-Adaptive Transformer

2019/10/22 by Maha Elbayad, Jiatao Gu, Elbayad, Maha +5 · 24 citations
Computer Science · #Neural Networks and Applications #Time Series Analysis and Forecasting #Image and Signal Denoising Methods

paper · pdf · doi:10.48550/arxiv.1910.10073

Abstract

State of the art sequence-to-sequence models for large scale tasks perform a fixed number of computations for each input sequence regardless of whether it is easy or hard to process. In this paper, we train Transformer models which can make output predictions at different stages of the network and we investigate different ways to predict how much computation is required for a particular sequence. Unlike dynamic computation in Universal Transformers, which applies the same set of layers iteratively, we apply different layers at every step to adjust both the amount of computation as well as the model capacity. On IWSLT German-English translation our approach matches the accuracy of a well tuned baseline Transformer while using less than a quarter of the decoder layers.

Cited by

• Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction
• Dynamic Rebatching for Efficient Early-Exit Inference with DREX
• How Smoothing is N-simplicial Attention?
• Generative Recursive Reasoning
• Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
• Dynamic Parameterization Is Not Dynamic Inference
• Depth-Adaptive Graph Recurrent Network for Text Classification
• Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models
• BitSkip: An Empirical Analysis of Quantization and Early Exit Composition
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• Fast Inference via Hierarchical Speculative Decoding
• Safe and Efficient In-Context Learning via Risk Control
• Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs
• Tapered Language Models
• MeSH: Memory-as-State-Highways for Recursive Transformers
• Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
• Entropy After ⟨ /Think ⟩ for reasoning model early exiting
• Learning to Ponder: Adaptive Reasoning in Latent Space
• Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliability
• DynaNav: Dynamic Feature and Layer Selection for Efficient Visual Navigation
• Looped Transformers with Source-Centered State Evolution
• ADMP-GNN: Adaptive Depth Message Passing GNN
• ExpertSim: Fast Particle Detector Simulation Using Mixture-of-Generative-Experts
• Formal Algorithms for Model Efficiency

Related

• Rewiring the Transformer with Depth-Wise LSTMs
• Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
• Universal Approximation Theorem for a Single-Layer Transformer
• Deep Transformers with Latent Depth
• Introduction to Sequence Modeling with Transformers