TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models

2025/06/15 by Yang Dai, Oubo Ma, Dai, Yang +15
Computer Science · #Action (physics) #Adversarial Robustness in Machine Learning #Backdoor #FOS: Computer and information sciences #Formal Methods in Verification #Machine Learning (cs.LG) #Reinforcement learning #Scalability #Sequence (biology) #Trajectory

paper · pdf · doi:10.48550/arxiv.2506.12815

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/06/15 · openalex created_date 2025/10/14 · openalex updated_date 2026/08/05

Abstract

Recent advances in Trajectory Optimization (TO) models have achieved remarkable success in offline reinforcement learning. However, their vulnerabilities against backdoor attacks are poorly understood. We find that existing backdoor attacks in reinforcement learning are based on reward manipulation, which are largely ineffective against the TO model due to its inherent sequence modeling nature. Moreover, the complexities introduced by high-dimensional action spaces further compound the challenge of action manipulation. To address these gaps, we propose TrojanTO, the first action-level backdoor attack against TO models. TrojanTO employs alternating training to enhance the connection between triggers and target actions for attack effectiveness. To improve attack stealth, it utilizes precise poisoning via trajectory filtering for normal performance and batch poisoning for trigger consistency. Extensive evaluations demonstrate that TrojanTO effectively implants backdoor attacks across diverse tasks and attack objectives with a low attack budget (0.3% of trajectories). Furthermore, TrojanTO exhibits broad applicability to DT, GDT, and DC, underscoring its scalability across diverse TO model architectures.

Citations

• TooBadRL: Trigger Optimization to Boost Effectiveness of Backdoor Attacks on Deep Reinforcement Learning
• Machine Learning Models Have a Supply Chain Problem
• UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learning
• BLAST: A Stealthy Backdoor Leverage Attack against Cooperative Multi-Agent Deep Reinforcement Learning based Systems
• RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviors
• Doubly Mild Generalization for Offline Reinforcement Learning
• Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
• Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editing
• A Spatiotemporal Stealthy Backdoor Attack against Cooperative Multi-Agent Deep Reinforcement Learning
• Knowledge Mechanisms in Large Language Models: A Survey and Perspective
• Is poisoning a real threat to LLM alignment? Maybe more so than you think
• SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents
• Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?
• TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents
• Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data
• SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systems
• Reinforcement Unlearning
• BadRL: Sparse Targeted Backdoor Attack Against Reinforcement Learning
• Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Training
• Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learning
• GPT-4 Technical Report
• Graph Decision Transformer
• Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective
• On Transforming Reinforcement Learning by Transformer: The Development Trajectory
• Provable Defense against Backdoor Policies in Reinforcement Learning
• Adversarial Policies Beat Superhuman Go AIs
• BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets
• Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learning
• A Generalist Agent
• Locating and Editing Factual Associations in GPT
• PolicyCleanse: Backdoor Detection and Mitigation in Reinforcement Learning
• MetaFormer Is Actually What You Need for Vision
• MetaFormer is Actually What You Need for Vision
• Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning
• Poisoning Deep Reinforcement Learning Agents with In-Distribution Triggers
• Handcrafted Backdoors in Deep Neural Networks
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning
• Rewriting a Deep Generative Model
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Policy Poisoning in Batch Reinforcement Learning and Control
• Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agents
• Adversarial Policies: Attacking Deep Reinforcement Learning
• TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents
• Design of intentional backdoors in sequential models
• Off-Policy Deep Reinforcement Learning without Exploration
• Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering
• Boosting Adversarial Attacks with Momentum
• Deep Reinforcement Learning that Matters
• Attention Is All You Need
• Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
• TrojanEdit: Multimodal Backdoor Attack Against Image Editing Model

Related

• UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learning
• Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning
• Exposing Vulnerabilities in RL: A Novel Stealthy Backdoor Attack through Reward Poisoning
• Adversarial Inception Backdoor Attacks against Reinforcement Learning
• Adversarial Backdoor Attack by Naturalistic Data Poisoning on Trajectory Prediction in Autonomous Driving