Long Short-Term Memory-Networks for Machine Reading

2016/01/25 by Cheng, Jianpeng, Dong, Li, Lapata, Mirella · 9 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Neural and Evolutionary Computing (cs.NE)

paper · doi:10.48550/arxiv.1601.06733

Abstract

In this paper we address the question of how to render sequence-level networks better at handling structured input. We propose a machine reading simulator which processes text incrementally from left to right and performs shallow reasoning with memory and attention. The reader extends the Long Short-Term Memory architecture with a memory network in place of a single memory cell. This enables adaptive memory usage during recurrence with neural attention, offering a way to weakly induce relations among tokens. The system is initially designed to process a single sequence but we also demonstrate how to integrate it with an encoder-decoder architecture. Experiments on language modeling, sentiment analysis, and natural language inference show that our model matches or outperforms the state of the art.

Cited by

• The Epistemological Consequences of Large Language Models: Rethinking collective intelligence and institutional knowledge
• Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGI
• MTMed3D: A Multi-Task Transformer-Based Model for 3D Medical Imaging
• Learning Robust Satellite Attitude Dynamics with Physics-Informed Normalising Flow
• Time Series Forecasting Through the Lens of Dynamics
• The Origin of Self-Attention: Pairwise Affinity Matrices in Feature Selection and the Emergence of Self-Attention
• Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictions
• Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning
• A comparative study of deep learning and ensemble learning to extend the horizon of traffic forecasting

Related

• End-To-End Memory Networks
• Survey of reasoning using Neural networks
• Extending Memory for Language Modelling
• Titans: Learning to Memorize at Test Time
• Learning Operations on a Stack with Neural Turing Machines