Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals

2019/06/24 by Yunhan Huang, Quanyan Zhu, Huang, Yunhan +1 · 5 citations
Computer Science · Engineering · Social Sciences · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Crime, Illicit Activities, and Governance #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC) #Smart Grid Security and Resilience

paper · pdf · doi:10.48550/arxiv.1906.10571

openalex publication_date 2019/06/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper studies reinforcement learning (RL) under malicious falsification on cost signals and introduces a quantitative framework of attack models to understand the vulnerabilities of RL. Focusing on Q-learning, we show that Q-learning algorithms converge under stealthy attacks and bounded falsifications on cost signals. We characterize the relation between the falsified cost and the Q-factors as well as the policy learned by the learning agent which provides fundamental limits for feasible offensive and defensive moves. We propose a robust region in terms of the cost within which the adversary can never achieve the targeted policy. We provide conditions on the falsified cost which can mislead the agent to learn an adversary's favored policy. A numerical case study of water reservoir control is provided to show the potential hazards of RL in learning-based control systems and corroborate the results.

Citations

• Stochastic Approximation: A Dynamical Systems Viewpoint

Cited by

• Diffusion Guided Adversarial State Perturbations in Reinforcement Learning
• Learn to Exceed: Stereo Inverse Reinforcement Learning with Concurrent Policy Optimization
• Provably Invincible Adversarial Attacks on Reinforcement Learning Systems: A Rate-Distortion Information-Theoretic Approach
• Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learning
• Adaptive Reward-Poisoning Attacks against Reinforcement Learning

Related

• Manipulating Reinforcement Learning: Poisoning Attacks on Cost Signals
• Reinforcement Learning for Linear Quadratic Control is Vulnerable Under Cost Manipulation
• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
• Provably Invincible Adversarial Attacks on Reinforcement Learning Systems: A Rate-Distortion Information-Theoretic Approach
• Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks