Highway Networks

2015/05/03 by Rupesh K. Srivastava, Rupesh Kumar Srivastava, Klaus Greff +4 · 1 voice · 256 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Business #Computer science #Stochastic Gradient Optimization Techniques #acm:68T01 #cs.LG #cs.NE #msc:68T01

paper · pdf · doi:10.48550/arxiv.1505.00387

published in arXiv (Cornell University) (Cornell University) · 6 pages, 2 figures. Presented at ICML 2015 Deep Learning workshop. Full paper is at arXiv:1507.06228

openalex publication_date 2015/05/03 · arxiv created 2015/11/03 · arxiv updated 2015/11/04 · openalex created_date 2024/04/10 · openalex updated_date 2026/08/05

Abstract

There is plenty of theoretical and empirical evidence that depth of neural networks is a crucial ingredient for their success. However, network training becomes more difficult with increasing depth and training of very deep networks remains an open problem. In this extended abstract, we introduce a new architecture designed to ease gradient-based training of very deep networks. We refer to networks with this architecture as highway networks, since they allow unimpeded information flow across several layers on "information highways". The architecture is characterized by the use of gating units which learn to regulate the flow of information through a network. Highway networks with hundreds of layers can be trained directly using stochastic gradient descent and with a variety of activation functions, opening up the possibility of studying extremely deep and efficient architectures.

Citations

• Training Very Deep Networks

Cited by

• Pretraining Recurrent Networks without Recurrence
• Glyce: Glyph-vectors for Chinese Character Representations
• mHC-GNN: Manifold-Constrained Hyper-Connections for Graph Neural Networks
• xHC: Expanded Hyper-Connections
• Fast weight programming and linear transformers: from machine learning to neurobiology
• Attention Residuals
• Greedy dynamical meta-learning
• Deep Delta Learning
• On Memory: A comparison of memory mechanisms in world models
• QL-LSTM: A Parameter-Efficient LSTM for Stable Long-Sequence Modeling
• LSRS: Latent Scale Rejection Sampling for Visual Autoregressive Modeling
• Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis
• Dual-Pathway Fusion of EHRs and Knowledge Graphs for Predicting Unseen Drug-Drug Interactions
• MeSH: Memory-as-State-Highways for Recursive Transformers
• Recurrence-Complete Frame-based Action Models
• LUMA: Low-Dimension Unified Motion Alignment with Dual-Path Anchoring for Text-to-Motion Diffusion Model
• Sig2Model: A Boosting-Driven Model for Updatable Learned Indexes
• Interpretable Representation via LLM-Driven Generative Disentanglement for Local-Life Service Recommendation
• mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations
• SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation
• Excavate the potential of Single-Scale Features: A Decomposition Network for Water-Related Optical Image Enhancement
• Neural Networks with Orthogonal Jacobian
• The Vanishing Gradient Problem for Stiff Neural Differential Equations
• Foundations and Models in Modern Computer Vision: Key Building Blocks in Landmark Architectures
• Graph neural networks for residential location choice: connection to classical logit models
• MVIAnalyzer: A Holistic Approach to Analyze Missing Value Imputation
• Auto-Compressing Networks
• Neural Inhibition Improves Dynamic Routing and Mixture of Experts
• The Generalist Brain Module: Module Repetition in Neural Networks in Light of the Minicolumn Hypothesis
• Busting the Paper Ballot: Voting Meets Adversarial Machine Learning
• Don't Pay Attention
• Sparsified State-Space Models are Efficient Highway Networks
• CrosGrpsABS: Cross-Attention over Syntactic and Semantic Graphs for Aspect-Based Sentiment Analysis in a Low-Resource Language
• Net-Zero: A Comparative Study on Neural Network Design for Climate-Economic PDEs Under Uncertainty
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
• Training Very Deep Networks
• Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
• Representation Learning for Natural Language Processing
• Scattering Networks for Hybrid Representation Learning
• Unsupervised Deep Representation Learning and Few-Shot Classification of PolSAR Images
• AdaptoVision: A Multi-Resolution Image Recognition Model for Robust and Scalable Classification

Discussions

• Highway Networks [hn, 24 points, 2 comments]

Related

• Training Very Deep Networks
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Highway State Gating for Recurrent Highway Networks: improving information flow through time
• Graph Highway Networks
• Faster Training of Very Deep Networks Via p-Norm Gates