Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

2016/04/13 by Qianli Liao, Tomaso Poggio, Liao, Qianli +1 · 35 citations
Computer Science · Neuroscience · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Neural dynamics and brain function #Visual Attention and Saliency Detection #Visual perception and processing mechanisms #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1604.03640

This version was written in Sept. 2016. For April 2016 version see v1 below

openalex publication_date 2016/04/13 · arxiv created 2020/12/31 · arxiv updated 2021/01/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a special type of shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers. A direct implementation of such a RNN, although having orders of magnitude fewer parameters, leads to a performance similar to the corresponding ResNet. We propose 1) a generalization of both RNN and ResNet architectures and 2) the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex. We demonstrate the effectiveness of the architectures by testing them on the CIFAR-10 and ImageNet dataset.

Citations

• How Important is Weight Symmetry in Backpropagation?
• Long Short-Term Memory
• ImageNet classification with deep convolutional neural networks
• How Deep is the Feature Analysis underlying Rapid Visual Categorization?
• Learning Functions: When Is Deep Better Than Shallow
• Deep Networks with Stochastic Depth
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Block-Recurrent Dynamics in Vision Transformers
• Hierarchical Reasoning Models: Perspectives and Misconceptions
• Looped Transformers with Source-Centered State Evolution
• Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation
• Simulated Cortical Magnification Supports Self-Supervised Object Learning
• Deep Learning Empowered Super-Resolution: A Comprehensive Survey and Future Prospects
• Incorporating Visual Cortical Lateral Connection Properties into CNN: Recurrent Activation and Excitatory-Inhibitory Separation
• Transfer entropy-based feedback improves performance in artificial neural networks
• Image Segmentation by Iterative Inference from Conditional Score Estimation
• Can Active Memory Replace Attention?
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Recurrent networks improve neural response prediction and provide insights into underlying cortical circuits
• Learning Latent Causal Structures with a Redundant Input Neural Network
• Hierarchical Predictive Coding Models in a Deep-Learning Framework
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Review: Ordinary Differential Equations For Deep Learning
• One Size Fits Many: Column Bundle for Multi-X Learning
• Task-Driven Convolutional Recurrent Models of the Visual System
• Learning Normalized Inputs for Iterative Estimation in Medical Image Segmentation
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• Hard Encoding of Physics for Learning Spatiotemporal Dynamics
• Back-Projection Pipeline
• ItNet: iterative neural networks with small graphs for accurate, efficient and anytime semantic segmentation
• Spiking Deep Residual Network
• Gated Path Planning Networks
• Convolutional Networks with Dense Connectivity
• Differential equations as models of deep neural networks
• Recurrent Connectivity Aids Recognition of Partly Occluded Objects
• Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks
• Image Super-Resolution via Dual-State Recurrent Networks
• A Dynamically Controlled Recurrent Neural Network for Modeling Dynamical Systems
• Disentangling neural mechanisms for perceptual grouping
• Towards WARSHIP: Combining Components of Brain-Inspired Computing of RSH for Image Super Resolution
• Recurrent Batch Normalization
• Identity Connections in Residual Nets Improve Noise Stability
• Better artificial intelligence does not mean better models of biology

Related

• Task-Driven Convolutional Recurrent Models of the Visual System
• A shallow residual neural network to predict the visual cortex response
• Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs
• Interpretation of ResNet by Visualization of Preferred Stimulus in Receptive Fields
• Improving Neural Predictivity in the Visual Cortex with Gated Recurrent Connections