Quasi-Recurrent Neural Networks

2016/11/05 by James Bradbury, Bradbury, James, Stephen Merity +5 · 56 citations
Computer Science · #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1611.01576

Submitted to conference track at ICLR 2017

arxiv created 2016/11/21 · arxiv updated 2016/11/22

Abstract

Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep's computation on the previous timestep's output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks.

Citations

• Densely Connected Convolutional Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Cited by

• Why Are Linear RNNs More Parallelizable?
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Deep Voice: Real-time Neural Text-to-Speech
• A Lightweight Recurrent Network for Sequence Modeling
• Equilibrated Recurrent Neural Network: Neuronal Time-Delayed Self-Feedback Improves Accuracy and Stability
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech
• Cross Temporal Recurrent Networks for Ranking Question Answer Pairs
• Evaluating neural network explanation methods using hybrid documents and morphological agreement
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• Parallelizing Linear Recurrent Neural Nets Over Sequence Length
• Analysing Dropout and Compounding Errors in Neural Language Models
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
• Progress and Tradeoffs in Neural Language Models
• Anticipating Traffic Accidents with Adaptive Loss and Large-scale Incident DB
• A Survey on 3D Skeleton-Based Action Recognition Using Learning Method
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
• Learning to Search in Long Documents Using Document Structure
• STAGNet: A Spatio-Temporal Graph and LSTM Framework for Accident Anticipation
• Single Stream Parallelization of Recurrent Neural Networks for Low Power and Fast Inference
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• Alternating Synthetic and Real Gradients for Neural Language Modeling
• ME³-BEV: Mamba-Enhanced Deep Reinforcement Learning for End-to-End Autonomous Driving with BEV-Perception
• The Implicit and Explicit Regularization Effects of Dropout
• The neural architecture of language: Integrative modeling converges on predictive processing
• Learning to Attend via Word-Aspect Associative Fusion for Aspect-based Sentiment Analysis
• Simplifying Neural Machine Translation with Addition-Subtraction Twin-Gated Recurrent Networks
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• Recurrent Attention Unit
• Convolutional Sequence to Sequence Learning
• Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN
• Weighted Transformer Network for Machine Translation
• Modeling Vocabulary for Big Code Machine Learning
• mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling
• Detecting Hostile Posts using Relational Graph Convolutional Network
• Deep Learning Brasil -- NLP at SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets
• Compressive Transformers for Long-Range Sequence Modelling
• Regularizing and Optimizing LSTM Language Models
• Sequential-Parallel Duality in Prefix Scannable Models
• Multi-axis Attentive Prediction for Sparse EventData: An Application to Crime Prediction
• Distilling Large Language Models into Tiny and Effective Students using pQRNN
• Mixture of Expert/Imitator Networks: Scalable Semi-supervised Learning Framework
• 2kenize: Tying Subword Sequences for Chinese Script Conversion
• Recognizing Video Events with Varying Rhythms
• Physics-inspired Energy Transition Neural Network for Sequence Learning
• Semi-tied Units for Efficient Gating in LSTM and Highway Networks
• StateSpaceDiffuser: Bringing Long Context to Diffusion World Models
• Adaptively Truncating Backpropagation Through Time to Control Gradient Bias
• NAUTILUS: a Versatile Voice Cloning System
• Machine Translation of Low-Resource Spoken Dialects: Strategies for Normalizing Swiss German
• Rethinking Full Connectivity in Recurrent Neural Networks
• SoPa: Bridging CNNs, RNNs, and Weighted Finite-State Machines
• Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
• Low-Resource Neural Machine Translation Using Recurrent Neural Networks and Transfer Learning: A Case Study on English-to-Igbo
• Rumour Detection via Zero-shot Cross-lingual Transfer Learning
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
• Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM

Related

• Sliced Recurrent Neural Networks
• Fast-Slow Recurrent Neural Networks
• Thick-Net: Parallel Network Structure for Sequential Modeling
• Going Wider: Recurrent Neural Network With Parallel Cells
• Gradual Learning of Recurrent Neural Networks