Optimal Regularization Under Uncertainty: Distributional Robustness and Convexity Constraints

2025/10/03 by Oscar Leong, Leong, Oscar, Eliza O’Reilly +3
Computer Science · Decision Sciences · #Distributed Sensor Networks and Detection Algorithms #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (stat.ML) #Metric Geometry (math.MG) #Multi-Criteria Decision Making #Optimization and Control (math.OC) #Risk and Portfolio Optimization #Statistics Theory (math.ST)

paper · pdf · doi:10.48550/arxiv.2510.03464

openalex publication_date 2025/10/03 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

Regularization is a central tool for addressing ill-posedness in inverse problems and statistical estimation, with the choice of a suitable penalty often determining the reliability and interpretability of downstream solutions. While recent work has characterized optimal regularizers for well-specified data distributions, practical deployments are often complicated by distributional uncertainty and the need to enforce structural constraints such as convexity. In this paper, we introduce a framework for distributionally robust optimal regularization, which identifies regularizers that remain effective under perturbations of the data distribution. Our approach leverages convex duality to reformulate the underlying distributionally robust optimization problem, eliminating the inner maximization and yielding formulations that are amenable to numerical computation. We show how the resulting robust regularizers interpolate between memorization of the training distribution and uniform priors, providing insights into their behavior as robustness parameters vary. For example, we show how certain ambiguity sets, such as those based on the Wasserstein-1 distance, naturally induce regularity in the optimal regularizer by promoting regularizers with smaller Lipschitz constants. We further investigate the setting where regularizers are required to be convex, formulating a convex program for their computation and illustrating their stability with respect to distributional shifts. Taken together, our results provide both theoretical and computational foundations for designing regularizers that are reliable under model uncertainty and structurally constrained for robust deployment.

Citations

• Learning Difference-of-Convex Regularizers for Inverse Problems: A Flexible Framework with Theoretical Guarantees
• Towards optimal algorithms for the recovery of low-dimensional models with linear rates
• The Star Geometry of Critic-Based Regularizer Learning
• Weakly Convex Regularisers for Inverse Problems: Convergence of Critical Points and Primal-Dual Optimisation
• On Learning the Optimal Regularization Parameter in Inverse Problems
• Prior Mismatch and Adaptation in PnP-ADMM with a Nonconvex Convergence Analysis
• On Optimal Regularization Parameters via Bilevel Learning
• Optimal Regularization for a Data Source
• The out-of-sample prediction error of the square-root-LASSO and related estimators
• A theory of optimal convex regularization for low-dimensional recovery
• Learned convex regularizers for inverse problems
• Wasserstein Distributionally Robust Optimization: Theory and\n Applications in Machine Learning
• Invertible generative models for inverse problems: mitigating representation error and dataset bias
• Adversarial Regularizers in Inverse Problems
• Quantifying Distributional Model Risk via Optimal Transport
• Bilevel approaches for learning of variational imaging models
• Compressed Sensing off the Grid
• Exact Matrix Completion via Convex Optimization
• Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
• Stable Signal Recovery from Incomplete and Inaccurate Measurements
• Stable signal recovery from incomplete and inaccurate measurements
• An iterative thresholding algorithm for linear inverse problems with a sparsity constraint
• Nonlinear total variation based noise removal algorithms

Related

• Optimal Regularization for a Data Source
• Distributional Robustness and Regularization in Reinforcement Learning
• Regularization for Wasserstein Distributionally Robust Optimization
• Wasserstein distributionally robust optimization and its tractable regularization formulations
• Perturbation-Aware Distributionally Robust Optimization for Inverse Problems