Learning to Execute

2014/10/17 by Wojciech Zaremba, Ilya Sutskever, Zaremba, Wojciech +1 · 2 voices · 38 citations
Computer Science · #Machine Learning and Algorithms #Neural Networks and Applications #Topic Modeling #cs.AI #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1410.4615

openalex publication_date 2014/10/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recurrent Neural Networks (RNNs) with Long Short-Term Memory units (LSTM) are widely used because they are expressive and are easy to train. Our interest lies in empirically evaluating the expressiveness and the learnability of LSTMs in the sequence-to-sequence regime by training them to evaluate short computer programs, a domain that has traditionally been seen as too complex for neural networks. We consider a simple class of programs that can be evaluated with a single left-to-right pass using constant memory. Our main result is that LSTMs can learn to map the character-level representations of such programs to their correct outputs. Notably, it was necessary to use curriculum learning, and while conventional curriculum learning proved ineffective, we developed a new variant of curriculum learning that improved our networks' performance in all experimental conditions. The improved curriculum had a dramatic impact on an addition problem, making it possible to train an LSTM to add two 9-digit numbers with 99% accuracy.

Citations

• Recurrent Neural Network Regularization
• Sequence to Sequence Learning with Neural Networks

Cited by

• Gradient-Based Program Repair: Fixing Bugs in Continuous Program Spaces
• ShapeWorld - A new test methodology for multimodal language understanding
• Automatic Goal Generation for Reinforcement Learning Agents
• Doctor AI: Predicting Clinical Events via Recurrent Neural Networks
• Program Synthesis with Large Language Models
• SymMaP: Improving Computational Efficiency in Linear Solvers through Symbolic Preconditioning
• Graph2Seq: Graph to Sequence Learning with Attention-based Neural Networks
• Once Upon an Input: Reasoning via Per-Instance Program Synthesis
• Chain of Execution Supervision Promotes General Reasoning in Large Language Models
• Neural Execution of Graph Algorithms
• Advancing Routing-Awareness in Analog ICs Floorplanning
• Layer-Aware Influence for Online Data Valuation Estimation
• Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings
• Evaluating Large Language Models Trained on Code
• An empirical study on the limitation of Transformers in program trace generation
• Article citation study: Context enhanced citation sentiment detection
• Query-Efficient Imitation Learning for End-to-End Autonomous Driving
• MLBiNet: A Cross-Sentence Collective Event Detection Network
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Grammar as a Foreign Language
• Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Improving performance of recurrent neural network with relu nonlinearity
• Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning
• An Empirical Comparison of Syllabuses for Curriculum Learning
• Mastering Rate based Curriculum Learning
• TestWeaver: Execution-aware, Feedback-driven Regression Testing Generation with Large Language Models
• Real-time interactive sequence generation and control with Recurrent\n Neural Network ensembles
• Image Captioning with Deep Bidirectional LSTMs
• Order Matters: Sequence to sequence for sets
• Video Summarization with Long Short-term Memory
• Attend to You: Personalized Image Captioning with Context Sequence Memory Networks
• Beyond Short Snippets: Deep Networks for Video Classification
• Survey of reasoning using Neural networks
• Region Growing Curriculum Generation for Reinforcement Learning
• Learning Algorithmic Solutions to Symbolic Planning Tasks with a Neural Computer Architecture
• Protein Secondary Structure Prediction with Long Short Term Memory Networks

Discussions

• Learning to Execute [hn, 42 points, 1 comments]
• Learning to Execute [pdf] [hn, 16 points, 2 comments]

Related

• Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks
• Long Short-Term Memory
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network
• LSTM Networks Can Perform Dynamic Counting
• On Evaluating the Generalization of LSTM Models in Formal Languages