Depth-Adaptive Transformer

2019/10/22 by Maha Elbayad, Jiatao Gu, Elbayad, Maha +6 · 45 citations
Computer Science · #Image and Signal Denoising Methods #Neural Networks and Applications #Time Series Analysis and Forecasting #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.1910.10073

Published as a conference paper at ICLR 2020

arxiv created 2020/02/14 · arxiv updated 2020/02/18

Abstract

State of the art sequence-to-sequence models for large scale tasks perform a fixed number of computations for each input sequence regardless of whether it is easy or hard to process. In this paper, we train Transformer models which can make output predictions at different stages of the network and we investigate different ways to predict how much computation is required for a particular sequence. Unlike dynamic computation in Universal Transformers, which applies the same set of layers iteratively, we apply different layers at every step to adjust both the amount of computation as well as the model capacity. On IWSLT German-English translation our approach matches the accuracy of a well tuned baseline Transformer while using less than a quarter of the decoder layers.

Cited by

• Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction
• Dynamic Rebatching for Efficient Early-Exit Inference with DREX
• How Smoothing is N-simplicial Attention?
• Generative Recursive Reasoning
• Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
• Dynamic Parameterization Is Not Dynamic Inference
• Depth-Adaptive Graph Recurrent Network for Text Classification
• Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models
• BitSkip: An Empirical Analysis of Quantization and Early Exit Composition
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• Fast Inference via Hierarchical Speculative Decoding
• Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting
• Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs
• Tapered Language Models
• MeSH: Memory-as-State-Highways for Recursive Transformers
• Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
• Entropy After ⟨ /Think ⟩ for reasoning model early exiting
• Learning to Ponder: Adaptive Reasoning in Latent Space
• Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliability
• DynaNav: Dynamic Feature and Layer Selection for Efficient Visual Navigation
• Looped Transformers with Source-Centered State Evolution
• ADMP-GNN: Adaptive Depth Message Passing GNN
• ExpertSim: Fast Particle Detector Simulation Using Mixture-of-Generative-Experts
• Formal Algorithms for Model Efficiency
• Dual-view Molecule Pre-training
• Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models
• Change of Thought: Adaptive Test-Time Computation
• FastFLUX: Pruning FLUX with Block-wise Replacement and Sandwich Training
• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
• Learning to Skip the Middle Layers of Transformers
• AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism
• Image Recognition with Online Lightweight Vision Transformer: A Survey
• Spatial Knowledge Graph-Guided Multimodal Synthesis
• FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation
• System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts
• Balanced and Elastic End-to-end Training of Dynamic LLMs
• Current Limitations of Language Models: What You Need is Retrieval
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
• DYNAMAX: Dynamic computing for Transformers and Mamba based architectures
• Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents
• River-LLM: Large Language Model Seamless Exit Based on KV Share
• Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity
• EPSILON: Adaptive Fault Mitigation in Approximate Deep Neural Network using Statistical Signatures
• Efficient Contrastive Decoding with Probabilistic Hallucination Detection - Mitigating Hallucinations in Large Vision Language Models -
• MACRO: Markov Chain Routing of Transformer Layers

Related

• Rewiring the Transformer with Depth-Wise LSTMs
• Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
• Universal Approximation Theorem for a Single-Layer Transformer
• Deep Transformers with Latent Depth
• Introduction to Sequence Modeling with Transformers