Robustness to Model Approximation, Model Learning From Data, and Sample Complexity in Wasserstein Regular MDPs

2024/10/18 by Yichen Zhou, Yanglei Song, Zhou, Yichen +3 · 2 citations
Computer Science · #Adversarial Robustness in Machine Learning #cs.SY #eess.SY #math.OC

paper · pdf · doi:10.48550/arxiv.2410.14116

openalex publication_date 2024/10/18 · openalex created_date 2024/11/02 · openalex updated_date 2026/07/28

Abstract

The paper studies the robustness properties of discrete-time stochastic optimal control under Wasserstein model approximation for both discounted-cost and average-cost criteria. Specifically, we study the performance loss when applying an optimal policy designed for an approximate model to the true dynamics compared with the optimal cost for the true model under the sup-norm-induced metric, and relate it to the Wasserstein-1 distance between the approximate and true transition kernels. A primary motivation of this analysis is empirical model learning, as well as empirical noise distribution learning, where Wasserstein convergence holds under mild conditions but stronger convergence criteria, such as total variation, may not. We discuss applications of the results to the disturbance estimation problem, where sample complexity bounds are given, and also to a general empirical model learning approach, obtained under either Markov or i.i.d. learning settings.

Cited by

• Quantizer Design for Finite Model Approximations, Model Learning, and Quantized Q-Learning for MDPs with Unbounded Spaces
• Sensitivity of Filter Kernels and Robustness Bounds to Transition and Measurement Kernel Perturbations in Partially Observable Stochastic Control

Related

• Robustness to Incorrect Models and Data-Driven Learning in Average-Cost Optimal Stochastic Control
• Robustness to incorrect system models in stochastic control
• Finite-Sample Guarantees for Wasserstein Distributionally Robust Optimization: Breaking the Curse of Dimensionality
• Wasserstein Distributionally Robust Control of Partially Observable Linear Systems: Tractable Approximation and Performance Guarantee
• Estimating processes in adapted Wasserstein distance