On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization

2016/07/19 by Stephen Jay Gould, Gould, Stephen, Basura Fernando +9 · 8 citations
Computer Science · Engineering · Mathematics · #Advanced Numerical Analysis Techniques #Advanced Optimization Algorithms Research #Computational Geometry and Mesh Generation #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Mathematics #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.1607.05447

openalex publication_date 2016/07/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Some recent works in machine learning and computer vision involve the solution of a bi-level optimization problem. Here the solution of a parameterized lower-level problem binds variables that appear in the objective of an upper-level problem. The lower-level problem typically appears as an argmin or argmax optimization problem. Many techniques have been proposed to solve bi-level optimization problems, including gradient descent, which is popular with current end-to-end learning approaches. In this technical report we collect some results on differentiating argmin and argmax optimization problems with and without constraints and provide some insightful motivating examples.

Citations

• Deep learning in neural networks: An overview

Cited by

• APT: Affine Prototype-Timestamp For Time Series Forecasting Under Distribution Shift
• Scalable Principal-Agent Contract Design via Gradient-Based Optimization
• Differentiable Model Predictive Control on the GPU
• Proximal Mapping for Deep Regularization
• A Neural Network Architecture to Learn Explicit MPC Controllers from Data
• Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization
• Debiasing a First-order Heuristic for Approximate Bi-level Optimization
• Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

Related

• Exploring the Generalization Capabilities of AID-based Bi-level Optimization
• Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond
• Constrained Bi-Level Optimization: Proximal Lagrangian Value function Approach and Hessian-free Algorithm
• Gradient-based Bi-level Optimization for Deep Learning: A Survey
• Optimization Methods in Deep Learning: A Comprehensive Overview