Duality and Policy Evaluation in Distributionally Robust Bayesian Diffusion Control

2025/06/24 by José Blanchet, Blanchet, Jose, Cheng, Jiayi +3
Decision Sciences · Computer Science · Engineering · #Advanced Bandit Algorithms Research #Reinforcement Learning in Robotics #Advanced Control Systems Optimization

paper · pdf · doi:10.48550/arxiv.2506.19294

Abstract

We study diffusion control problems under parameter uncertainty. Controllers based on plug-in estimation can be brittle due to potential distribution shifts. Bayesian control with a prior on the parameters offers a formulation with beliefs about such shifts. However, as with any Bayesian model, the prior may be misspecified. To mitigate misspecification and reduce over-pessimism compared to classical robust control approaches (e.g. \citethansen2008robustness), we propose a distributionally robust Bayesian control (DRBC) formulation in which an adversary perturbs the prior within a divergence neighborhood of a baseline prior. We develop a strong duality result that reduces the distributionally robust prior evaluation to a low-dimensional optimization and yields a practical simulation-based policy evaluation and learning procedure with structured policy parameterizations. We validate the efficiency of the algorithm on a synthetic linear-quadratic control example and real-data portfolio selection.

Citations

• Bayesian Distributionally Robust Merton Problem with Nonlinear Wasserstein Projections
• A Globally Optimal Portfolio for m-Sparse Sharpe Ratio Maximization
• Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
• Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms
• Distributionally Robust Optimization and Robust Statistics
• On the Foundation of Distributionally Robust Reinforcement Learning
• Data-Dependent Bounds for Online Portfolio Selection Without Lipschitzness and Smoothness
• A Finite Sample Complexity Bound for Distributionally Robust Q-learning
• Optimal randomized multilevel Monte Carlo for repeatedly nested expectations
• The continuous-time pre-commitment KMM problem in incomplete markets
• Policy Gradient Method For Robust Reinforcement Learning
• Robust Multi-Objective Bayesian Optimization Under Input Noise
• Bayesian Distributionally Robust Optimization
• Sinkhorn Distributionally Robust Optimization
• Statistical Analysis of Wasserstein Distributionally Robust Estimators
• Large-Scale Methods for Distributionally Robust Optimization
• Distributionally Robust Batch Contextual Bandits
• Distributionally Robust Bayesian Optimization
• Distributionally Robust Counterfactual Risk Minimization
• Unbiased Multilevel Monte Carlo: Stochastic Optimization, Steady-state Simulation, Quantiles, and Other Applications
• Learning Models with Uniform Performance via Distributionally Robust\n Optimization
• Efficient Online Portfolio with Logarithmic Regret
• Distributionally Robust Mean-Variance Portfolio Selection with Wasserstein Distances
• Quantifying Distributional Model Risk via Optimal Transport
• An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedback
• Optimal rates for zero-order convex optimization: the power of two\n function evaluations

Related

• Distributionally Robust Policy Learning under Concept Drifts
• Robustness of Stochastic Optimal Control to Approximate Diffusion Models under Several Cost Evaluation Criteria
• Learning Optimal Distributionally Robust Stochastic Control in Continuous State Spaces
• Distributionally Robust Gaussian Process Regression and Bayesian Inverse Problems
• Stochastic Optimal Control with Side Information and Bayesian Learning