Learning long-term dependencies with gradient descent is difficult

1994/03/01 by Y. Bengio, P. Simard, P. Frasconi · 100 citations

paper · doi:10.1109/72.279181

Cited by

• Reactive Transformer (RxT) -- Stateful Real-Time Processing for Event-Driven Reactive Language Models
• HRM-Text: Efficient Pretraining Beyond Scaling
• Human-AI Collaborative Uncertainty Quantification
• Dopamine-driven synaptic credit assignment in neural networks
• Extreme value forecasting using relevance-based data augmentation with deep learning models
• Weights initialization of neural networks for function approximation
• Artificial Hippocampus Networks for Efficient Long-Context Modeling
• HTMformer: Hybrid Time and Multivariate Transformer for Time Series Forecasting
• Gather-Scatter Mamba: Accelerating Propagation with Efficient State Space Model
• Identifying Information-Transfer Nodes in a Recurrent Neural Network Reveals Dynamic Representations
• Room Impulse Response Prediction with Neural Networks: From Energy Decay Curves to Perceptual Validation
• Deep Learning-Based Prediction of Energy Decay Curves from Room Geometry and Material Properties
• Who invented deep residual learning?
• PALQO: Physics-informed Model for Accelerating Large-scale Quantum Optimization
• Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications
• The Topological Trouble With Transformers
• Neural Network Based Framework for Passive Intermodulation Cancellation in MIMO Systems
• Deep Learning Inductive Biases for fMRI Time Series Classification during Resting-state and Movie-watching
• M4GN: Mesh-based Multi-segment Hierarchical Graph Network for Dynamic Simulations
• AquaCast: Urban Water Dynamics Forecasting with Precipitation-Informed Multi-Input Transformer
• Measuring General Associations in Time Series: An Adaptation and Empirical Evaluation of the CODEC Coefficient in Determining Autoregressive Dynamics
• Uncertainty-Aware Neural Networks for Fuzzy Dark Matter Model Selection from \texorpdfstringx_{\rm HI}x_HI Measurements
• Deep Learning for Conditional Asset Pricing Models
• A Brain-Inspired Gating Mechanism Unlocks Robust Computation in Spiking Neural Networks
• Motor Imagery EEG Signal Classification Using Minimally Random Convolutional Kernel Transform and Hybrid Deep Learning
• A constrained optimization approach to nonlinear system identification through simulation error minimization
• Deep Residual Echo State Networks: exploring residual orthogonal connections in untrained Recurrent Neural Networks
• C¹-approximation with rational functions and rational neural networks
• Revisiting associative recall in modern recurrent models
• Frozen in Time: Parameter-Efficient Time Series Transformers via Reservoir-Induced Feature Expansion and Fixed Random Dynamics
• Wormhole Dynamics in Deep Neural Networks
• Autoregressive Typical Thermal States
• Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification
• The Role of Entanglement in Quantum Reservoir Computing with Coupled Kerr Nonlinear Oscillators
• Lameness detection in dairy cows using pose estimation and bidirectional LSTMs
• Reinforcement learning in densely recurrent biological networks
• Chi-Geometry: A Library for Benchmarking Chirality Prediction of GNNs
• Neural Networks with Orthogonal Jacobian
• The Vanishing Gradient Problem for Stiff Neural Differential Equations
• Forecasting NCAA Basketball Outcomes with Deep Learning: A Comparative Study of LSTM and Transformer Models
• Hebbian Memory-Augmented Recurrent Networks: Engram Neurons in Deep Learning
• Spatial-Temporal Reinforcement Learning for Network Routing with Non-Markovian Traffic
• Reinforced Embodied Active Defense: Exploiting Adaptive Interaction for Robust Visual Perception in Adversarial 3D Environments
• Quantum Boltzmann Machines using Parallel Annealing for Medical Image Classification
• Physics-guided gated recurrent units for inversion-based feedforward control
• Global weather forecasting through deep learning: applications and challenges
• A Wavelength-Aware Unsupervised Learning Approach for Large, Multicolor, Photometric Surveys
• The Serial Scaling Hypothesis
• Spatiotemporal deep learning models for detection of rapid intensification in cyclones
• Data-Driven Dimensional Synthesis of Diverse Planar Four-bar Function Generation Mechanisms via Direct Parameterization
• Data-Driven Reconstruction and Characterization of Stochastic Dynamics via Dynamical Mode Decomposition
• Introduction to Quantum Error Correction with Stabilizer Codes
• RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
• mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling
• EvRWKV: A Continuous Interactive RWKV Framework for Effective Event-Guided Low-Light Image Enhancement
• LearnAFE: Circuit-Algorithm Co-design Framework for Learnable Audio Analog Front-End
• A Survey of Retentive Network
• Uncovering the Functional Roles of Nonlinearity in Memory
• A parametric activation function based on Wendland RBF
• Storm Surge in Color: RGB-Encoded Physics-Aware Deep Learning for Storm Surge Forecasting
• Over-squashing in Spatiotemporal Graph Neural Networks
• pLSTM: parallelizable Linear Source Transition Mark networks
• Transient Dynamics in Lattices of Differentiating Ring Oscillators
• A projection-based framework for gradient-free and parallel learning
• Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions
• Temporal horizons in forecasting: a performance-learnability trade-off
• Learning Optical Flow Field via Neural Ordinary Differential Equation
• MVAN: Multi-View Attention Networks for Fake News Detection on Social Media
• Temporal Chunking Enhances Recognition of Implicit Sequential Patterns
• Deep Learning Weather Models for Subregional Ocean Forecasting: A Case Study on the Canary Current Upwelling System
• State Estimation and Control of Dynamic Systems from High-Dimensional Image Data
• Physics-inspired Energy Transition Neural Network for Sequence Learning
• The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence
• Scalable, Symbiotic, AI and Non-AI Agent Based Parallel Discrete Event Simulations
• DeepConvContext: A Multi-Scale Approach to Timeseries Classification in Human Activity Recognition
• Revisiting Glorot Initialization for Long-Range Linear Recurrences
• Classification of assembly tasks combining multiple primitive actions using Transformers and xLSTMs
• SpectraLDS: Provable Distillation for Linear Dynamical Systems
• Leveraging Stochastic Depth Training for Adaptive Inference
• Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses
• Sparse Latent Factor Forecaster (SLFF) with Iterative Inference for Transparent Multi-Horizon Commodity Futures Prediction
• Accurate and Efficient Multivariate Time Series Forecasting via Offline Clustering
• Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oracles
• Brain-Inspired Quantum Neural Architectures for Pattern Recognition: Integrating QSNN and QLSTM
• Parallelized Hierarchical Connectome: A Spatiotemporal Recurrent Framework for Spiking State-Space Models
• Don't be lazy: CompleteP enables compute-efficient deep transformers
• ParalESN: Enabling parallel information processing in Reservoir Computing
• Block Circulant Adapter for Large Language Models
• Adversarial observations in probabilistic State-Space Models for robust Reinforcement Learning
• Leveraging Surplus Electricity: Profitability of Bitcoin Mining as a National Strategy in South Korea
• Joint Return and Risk Modeling with Deep Neural Networks for Portfolio Construction
• Hybrid Quantum Recurrent Neural Network For Remaining Useful Life Prediction
• Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregation
• Beyond Sequential Prediction: Learning Financial Market Dynamics in Volatile and Non-Stationary Environments through Sentiment-Conditioned Generative Modelling
• Improving Deep Knowledge Tracing via Gated Architectures and Adaptive Optimization
• Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access
• A Basic Evaluation of Neural Networks Trained with the Error Diffusion Learning Algorithm
• Deep Learning and Econometric Time Series Analysis: An Assessment of Daily Return Forecasts
• Classification-Based Analysis of Price Pattern Differences Between Cryptocurrencies and Stocks
• Generative System Dynamics in Recurrent Neural Networks
• Episodically adapted network-based controllers