Structural-RNN: Deep Learning on Spatio-Temporal Graphs

2015/11/17 by Ashesh Jain, Jain, Ashesh, Amir R. Zamir +5 · 1 voice · 14 citations
#cs.CV #cs.LG #cs.NE #cs.RO

paper · pdf · doi:10.48550/arxiv.1511.05298

Abstract

Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatio-temporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks~(RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.

Cited by

• Reverberation: Learning the Latencies Before Forecasting Trajectories
• Language-guided Recursive Spatiotemporal Graph Modeling for Video Summarization
• Root Cause Analysis of Hydrogen Bond Separation in Spatio-Temporal Molecular Dynamics using Causal Models
• How Does a Virtual Agent Decide Where to Look? -- Symbolic Cognitive Reasoning for Embodied Head Rotation
• LuKAN: A Kolmogorov-Arnold Network Framework for 3D Human Motion Prediction
• SpikeSTAG: Spatial-Temporal Forecasting via GNN-SNN Collaboration
• UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generation
• RelMap: Reliable Spatiotemporal Sensor Data Visualization via Imputative Spatial Interpolation
• GGMotion: Group Graph Dynamics-Kinematics Networks for Human Motion Prediction
• Stochastic Human Motion Prediction with Memory of Action Transition and Action Characteristic
• Temporal Continual Learning with Prior Compensation for Human Motion Prediction
• Evaluating Generative Vehicle Trajectory Models for Traffic Intersection Dynamics
• HUMOF: Human Motion Forecasting in Interactive Social Scenes
• Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward

Discussions

• Deep Learning on Spatio-temporal graphs [hn, 16 points, 0 comments]

Related

• Structured Sequence Modeling with Graph Convolutional Recurrent Networks
• Spatio-temporal Graph-RNN for Point Cloud Prediction
• Deep RNN Framework for Visual Sequential Applications
• DMS-GCN: Dynamic Mutiscale Spatiotemporal Graph Convolutional Networks for Human Motion Prediction
• Exploiting Spatio-Temporal Structure with Recurrent Winner-Take-All\n Networks