Sample Average Approximation for Black-Box VI

2023/04/13 by Javier Burroni, Burroni, Javier, Justin Domke +3 · 2 citations
Computer Science · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Metaheuristic Optimization Algorithms Research #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2304.06803

openalex publication_date 2023/04/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a novel approach for black-box VI that bypasses the difficulties of stochastic gradient ascent, including the task of selecting step-sizes. Our approach involves using a sequence of sample average approximation (SAA) problems. SAA approximates the solution of stochastic optimization problems by transforming them into deterministic ones. We use quasi-Newton methods and line search to solve each deterministic optimization problem and present a heuristic policy to automate hyperparameter selection. Our experiments show that our method simplifies the VI problem and achieves faster performance than existing methods.

Cited by

• AutoGD: Automatic Learning Rate Selection for Gradient Descent
• Online Robust Planning under Model Uncertainty: A Sample-Based Approach

Related

• Stochastic quasi-Newton with adaptive step lengths for large-scale problems
• The Black-Box Optimization Problem: Zero-Order Accelerated Stochastic Method via Kernel Approximation
• Validating Sample Average Approximation Solutions with Negatively Dependent Batches
• Adaptive Sequential SAA for Solving Two-stage Stochastic Linear Programs
• 0/1 Constrained Optimization Solving Sample Average Approximation for Chance Constrained Programming