Multiscale Deep Equilibrium Models

2020/06/15 by Shaojie Bai, Vladlen Koltun, Bai, Shaojie +3 · 10 citations
Computer Science · Engineering · Mathematics · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Reinforcement Learning in Robotics #Reservoir Engineering and Simulation Methods

paper · pdf · doi:10.48550/arxiv.2006.08656

openalex publication_date 2020/06/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a new class of implicit networks, the multiscale deep equilibrium model (MDEQ), suited to large-scale and highly hierarchical pattern recognition domains. An MDEQ directly solves for and backpropagates through the equilibrium points of multiple feature resolutions simultaneously, using implicit differentiation to avoid storing intermediate states (and thus requiring only O(1) memory consumption). These simultaneously-learned multi-resolution features allow us to train a single model on a diverse set of tasks and loss functions, such as using a single MDEQ to perform both image classification and semantic segmentation. We illustrate the effectiveness of this approach on two large-scale vision tasks: ImageNet classification and semantic segmentation on high-resolution images from the Cityscapes dataset. In both settings, MDEQs are able to match or exceed the performance of recent competitive computer vision models: the first time such performance and scale have been achieved by an implicit deep learning approach. The code and pre-trained models are at https://github.com/locuslab/mdeq .

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• On the limited memory BFGS method for large scale optimization
• Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix
• Backpropagation Applied to Handwritten Zip Code Recognition
• ImageNet classification with deep convolutional neural networks
• Wide Residual Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Universal Transformers
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• UNet++: A Nested U-Net Architecture for Medical Image Segmentation
• TensorFlow: A system for large-scale machine learning
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Structured Knowledge Distillation for Dense Prediction
• Deep High-Resolution Representation Learning for Visual Recognition
• Self-Training With Noisy Student Improves ImageNet Classification
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Closed-Loop Transformers: Autoregressive Modeling as Iterative Latent Equilibrium
• Gradient descent for deep equilibrium single-index models
• Selection, Reflection and Self-Refinement: Revisit Reasoning Tasks via a Causal Lens
• End-to-End Training of High-Dimensional Optimal Control with Implicit Hamiltonians via Jacobian-Free Backpropagation
• NEO-Grid: A Neural Approximation Framework for Optimization and Control in Distribution Grids
• Reversible Deep Equilibrium Models
• Deep Equilibrium Convolutional Sparse Coding for Hyperspectral Image Denoising
• Local Scale Equivariance with Latent Deep Equilibrium Canonicalizer
• DyCAF-Net: Dynamic Class-Aware Fusion Network
• Greedy and Random Broyden's Methods with Explicit Superlinear Convergence Rates in Nonlinear Equations

Related

• Stabilizing Equilibrium Models by Jacobian Regularization
• Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks
• Reversible Deep Equilibrium Models
• Local Scale Equivariance with Latent Deep Equilibrium Canonicalizer
• SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit models