Visualizing and Understanding Recurrent Networks

2015/06/05 by Andrej Karpathy, Justin Johnson, Karpathy, Andrej +3 · 2 voices · 27 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Topic Modeling #cs.CL #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1506.02078

changing style, adding references, minor changes to text

openalex publication_date 2015/06/05 · arxiv published 2015/06/05 · arxiv created 2015/11/17 · arxiv updated 2015/11/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recurrent Neural Networks (RNNs), and specifically a variant with Long Short-Term Memory (LSTM), are enjoying renewed interest as a result of successful applications in a wide range of machine learning problems that involve sequential data. However, while LSTMs provide exceptional results in practice, the source of their performance and their limitations remain rather poorly understood. Using character-level language models as an interpretable testbed, we aim to bridge this gap by providing an analysis of their representations, predictions and error types. In particular, our experiments reveal the existence of interpretable cells that keep track of long-range dependencies such as line lengths, quotes and brackets. Moreover, our comparative analysis with finite horizon n-gram models traces the source of the LSTM improvements to long-range structural dependencies. Finally, we provide analysis of the remaining errors and suggests areas for further study.

Citations

• On the difficulty of training Recurrent Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Long Short-Term Memory
• Deep Learning in Neural Networks: An Overview
• Sequence to Sequence Learning with Neural Networks

Cited by

• Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
• CURA: Size Isnt All You Need -- A Compact Universal Architecture for On-Device Intelligence
• Superposition in Graph Neural Networks
• Yet Unnoticed in LSTM: Binary Tree Based Input Reordering, Weight Regularization, and Gate Nonlinearization
• Towards Explainable Deep Clustering for Time Series Data
• Exploring Task Performance with Interpretable Models via Sparse Auto-Encoders
• Attribution assignment for deep-generative sequence models enables interpretability analysis using positive-only data
• Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization
• DeepConvContext: A Multi-Scale Approach to Timeseries Classification in Human Activity Recognition
• FastCAV: Efficient Computation of Concept Activation Vectors for Explaining Deep Neural Networks
• Learning the Enigma with Recurrent Neural Networks
• Recursive Recurrent Nets with Attention Modeling for OCR in the Wild
• Interpretable Recurrent Neural Networks Using Sequential Sparse Recovery
• Improving Distributed Representations of Tweets - Present and Future
• Deep Reinforcement Learning: An Overview
• Compressing Neural Language Models by Sparse Word Representations
• Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents
• Understanding and Controlling Memory in Recurrent Neural Networks
• Mutual Information Scaling and Expressive Power of Sequence Models
• Definitions, methods, and applications in interpretable machine learning
• Analyzing and Interpreting Convolutional Neural Networks in NLP
• Do RNN States Encode Abstract Phonological Processes?
• Sampling for Deep Learning Model Diagnosis (Technical Report)
• What do Neural Machine Translation Models Learn about Morphology?
• Self-Explaining Structures Improve NLP Models
• Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities
• A Model Zoo of Vision Transformers

Discussions

• Visualizing and understanding recurrent networks [hn, 3 points, 0 comments]
• Visualizing and Understanding Recurrent Networks [hn, 2 points, 0 comments]

Related

• A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
• Language Modeling through Long Term Memory Network
• Learning Longer Memory in Recurrent Neural Networks
• Higher Order Recurrent Neural Networks
• How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?