Recurrent Attentive Neural Process for Sequential Data

2019/10/17 by Shenghao Qin, Qin, Shenghao, Jiacheng Zhu +7 · 23 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Computer science #Context (archaeology) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Neural Networks and Applications #Permutation (music) #Process (computing) #Recurrent neural network #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1910.09323

published in arXiv (Cornell University) (Cornell University) · 12 pages, 6 figures, NeurIPS 2019 Workshop

arxiv created 2019/10/17 · openalex publication_date 2019/10/17 · arxiv updated 2019/10/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Neural processes (NPs) learn stochastic processes and predict the distribution of target output adaptively conditioned on a context set of observed input-output pairs. Furthermore, Attentive Neural Process (ANP) improved the prediction accuracy of NPs by incorporating attention mechanism among contexts and targets. In a number of real-world applications such as robotics, finance, speech, and biology, it is critical to learn the temporal order and recurrent structure from sequential data. However, the capability of NPs capturing these properties is limited due to its permutation invariance instinct. In this paper, we proposed the Recurrent Attentive Neural Process (RANP), or alternatively, Attentive Neural Process-RecurrentNeural Network(ANP-RNN), in which the ANP is incorporated into a recurrent neural network. The proposed model encapsulates both the inductive biases of recurrent neural networks and also the strength of NPs for modelling uncertainty. We demonstrate that RANP can effectively model sequential data and outperforms NPs and LSTMs remarkably in a 1D regression toy example as well as autonomous-driving applications.

Citations

• A Recurrent Latent Variable Model for Sequential Data
• Long Short-Term Memory
• Sequence to Sequence Learning with Neural Networks
• Thoughts on Massively Scalable Gaussian Processes
• Stochastic Variational Deep Kernel Learning
• The Statistical Recurrent Unit
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Probabilistic Recurrent State-Space Models
• Sequential Neural Processes
• Generative Temporal Models with Spatial Memory for Partially Observed Environments

Cited by

• Few-shot Learning for Time-series Forecasting

Related

• Recurrent Neural Processes
• Learning The Sequential Temporal Information with Recurrent Neural Networks
• Particle Filter Recurrent Neural Networks
• Attentive Neural Processes
• A Novel Framework for Learning Stochastic Representations for Sequence Generation and Recognition