Provable Defense against Backdoor Policies in Reinforcement Learning

2022/11/18 by Shubham Bharti, Bharti, Shubham Kumar, Xuezhou Zhang +5 · 4 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2211.10530

openalex publication_date 2022/11/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a provable defense mechanism against backdoor policies in reinforcement learning under subspace trigger assumption. A backdoor policy is a security threat where an adversary publishes a seemingly well-behaved policy which in fact allows hidden triggers. During deployment, the adversary can modify observed states in a particular way to trigger unexpected actions and harm the agent. We assume the agent does not have the resources to re-train a good policy. Instead, our defense mechanism sanitizes the backdoor policy by projecting observed states to a 'safe subspace', estimated from a small number of interactions with a clean (non-triggered) environment. Our sanitized policy achieves ε approximate optimality in the presence of triggers, provided the number of clean interactions is O((D)/((1-γ)⁴ ε²)) where γ is the discounting factor and D is the dimension of state space. Empirically, we show that our sanitization defense performs well on two Atari game environments.

Cited by

• On the Tension Between Optimality and Adversarial Robustness in Policy Optimization
• Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Driving
• Beyond Training-time Poisoning: Component-level and Post-training Backdoors in Deep Reinforcement Learning
• TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models

Related

• Cooperative Backdoor Attack in Decentralized Reinforcement Learning with Theoretical Guarantee
• Recover Triggered States: Protect Model Against Backdoor Attack in Reinforcement Learning
• PolicyCleanse: Backdoor Detection and Mitigation in Reinforcement Learning
• BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning
• Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks