Adaptive Reward-Poisoning Attacks against Reinforcement Learning

2020/03/27 by Xuezhou Zhang, Yuzhe Ma, Zhang, Xuezhou +5 · 34 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Mathematics · Medicine · Psychology · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Artificial intelligence #Cardiac electrophysiology and arrhythmias #Computer science #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Psychology #Receptor Mechanisms and Signaling #Reinforcement #Reinforcement learning #cs.AI #cs.CR #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2003.12613

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2020/03/27 · arxiv created 2020/06/22 · arxiv updated 2020/06/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In reward-poisoning attacks against reinforcement learning (RL), an attacker can perturb the environment reward r_t into r_t+δ_t at each step, with the goal of forcing the RL agent to learn a nefarious policy. We categorize such attacks by the infinity-norm constraint on δ_t: We provide a lower threshold below which reward-poisoning attack is infeasible and RL is certified to be safe; we provide a corresponding upper threshold above which the attack is feasible. Feasible attacks can be further categorized as non-adaptive where δ_t depends only on (s_t,a_t, s_t+1), or adaptive where δ_t depends further on the RL agent's learning process at time t. Non-adaptive attacks have been the focus of prior works. However, we show that under mild conditions, adaptive attacks can achieve the nefarious policy in steps polynomial in state-space size |S|, whereas non-adaptive attacks require exponential steps. We provide a constructive proof that a Fast Adaptive Attack strategy achieves the polynomial rate. Finally, we show that empirically an attacker can find effective reward-poisoning attacks using state-of-the-art deep RL techniques.

Citations

• Adversarial Attacks on Stochastic Bandits
• Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals
• Policy Poisoning in Batch Reinforcement Learning and Control

Cited by

• Optimal Perturbation Budget Allocation for Data Poisoning in Offline Reinforcement Learning
• Exposing Vulnerabilities in RL: A Novel Stealthy Backdoor Attack through Reward Poisoning
• Diffusion Guided Adversarial State Perturbations in Reinforcement Learning
• Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
• Provably Invincible Adversarial Attacks on Reinforcement Learning Systems: A Rate-Distortion Information-Theoretic Approach
• Density-Ratio Weighted Behavioral Cloning: Learning Control Policies from Corrupted Datasets
• Constrained Black-Box Attacks Against Multi-Agent Reinforcement Learning
• Policy Disruption in Reinforcement Learning:Adversarial Attack with Large Language Models and Critical State Identification
• Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning
• Robust Policy Gradient against Strong Data Corruption
• Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimization
• Robust In-Context Reinforcement Learning Under Reward Poisoning Attacks
• Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning
• Seven Security Challenges in Cross-domain Multi-agent LLM Systems
• Sequential Attacks on Kalman Filter-based Forward Collision Warning Systems
• The Sample Complexity of Teaching-by-Reinforcement on Q-Learning

Related

• Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments
• Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learning
• Defense Against Reward Poisoning Attacks in Reinforcement Learning
• Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning
• Disturbing Reinforcement Learning Agents with Corrupted Rewards