Adaptive Reward-Poisoning Attacks against Reinforcement Learning

2020/03/27 by Xuezhou Zhang, Yuzhe Ma, Zhang, Xuezhou +5 · 7 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Medicine · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Cardiac electrophysiology and arrhythmias #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Receptor Mechanisms and Signaling

paper · pdf · doi:10.48550/arxiv.2003.12613

openalex publication_date 2020/03/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In reward-poisoning attacks against reinforcement learning (RL), an attacker can perturb the environment reward r_t into r_t+δ_t at each step, with the goal of forcing the RL agent to learn a nefarious policy. We categorize such attacks by the infinity-norm constraint on δ_t: We provide a lower threshold below which reward-poisoning attack is infeasible and RL is certified to be safe; we provide a corresponding upper threshold above which the attack is feasible. Feasible attacks can be further categorized as non-adaptive where δ_t depends only on (s_t,a_t, s_t+1), or adaptive where δ_t depends further on the RL agent's learning process at time t. Non-adaptive attacks have been the focus of prior works. However, we show that under mild conditions, adaptive attacks can achieve the nefarious policy in steps polynomial in state-space size |S|, whereas non-adaptive attacks require exponential steps. We provide a constructive proof that a Fast Adaptive Attack strategy achieves the polynomial rate. Finally, we show that empirically an attacker can find effective reward-poisoning attacks using state-of-the-art deep RL techniques.

Citations

• Adversarial Attacks on Stochastic Bandits
• Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals
• Policy Poisoning in Batch Reinforcement Learning and Control

Cited by

• Optimal Perturbation Budget Allocation for Data Poisoning in Offline Reinforcement Learning
• Exposing Vulnerabilities in RL: A Novel Stealthy Backdoor Attack through Reward Poisoning
• Diffusion Guided Adversarial State Perturbations in Reinforcement Learning
• Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
• Provably Invincible Adversarial Attacks on Reinforcement Learning Systems: A Rate-Distortion Information-Theoretic Approach
• Density-Ratio Weighted Behavioral Cloning: Learning Control Policies from Corrupted Datasets
• Constrained Black-Box Attacks Against Multi-Agent Reinforcement Learning

Related

• Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments
• Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learning
• Defense Against Reward Poisoning Attacks in Reinforcement Learning
• Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning
• Disturbing Reinforcement Learning Agents with Corrupted Rewards