Robust In-Context Reinforcement Learning Under Reward Poisoning Attacks

2025/06/07 by Paulius Sasnauskas, Sasnauskas, Paulius, Yiğit Yalın +3
Computer Science · Medicine · Neuroscience · #Adversarial Robustness in Machine Learning #Cardiac electrophysiology and arrhythmias #Cryptography and Security (cs.CR) #EEG and Brain-Computer Interfaces #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2506.06891

openalex publication_date 2025/06/07 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

We study the corruption-robustness of in-context reinforcement learning (ICRL), focusing on the Decision-Pretrained Transformer (DPT, Lee et al., 2023). To address the challenge of reward poisoning attacks targeting the DPT, we propose a novel adversarial training framework, called Adversarially Trained DPT (AT-DPT). Our method simultaneously trains a population of attackers to minimize the true reward of the DPT by poisoning environment rewards, and a DPT model to infer optimal actions from the poisoned data. We evaluate the effectiveness of our approach against standard bandit algorithms, including robust baselines designed to handle reward contamination. Our results show that AT-DPT significantly outperforms them in bandit settings under a learned attacker, and generalizes to more complex environments such as adaptive attackers and MDPs. It shows promise in ICRL as a meta-RL approach to learning effective corruption-robust algorithms.

Citations

• In-Context Learning for Pure Exploration
• Filtering Learning Histories Enhances In-Context Reinforcement Learning
• Vintix: Action Model via In-Context Reinforcement Learning
• Robust Thompson Sampling Algorithms Against Reward Poisoning Attacks
• LLMs Are In-Context Bandit Reinforcement Learners
• Adversarially Robust Decision Transformer
• Exploring the Robustness of In-Context Learning with Noisy Labels
• Data Poisoning for In-context Learning
• Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning
• Emergence of In-Context Reinforcement Learning from Noise Distillation
• Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
• Supervised Pretraining Can Learn In-Context Reinforcement Learning
• Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks
• Structured State Space Models for In-Context Reinforcement Learning
• Decision Transformer under Random Frame Dropping
• Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attacks
• Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Poisoning Web-Scale Training Datasets is Practical
• A Tutorial on Meta-Reinforcement Learning
• In-context Reinforcement Learning with Algorithm Distillation
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
• Robust Reinforcement Learning using Offline Data
• Prompting Decision Transformer for Few-Shot Policy Generalization
• RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
• Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learning
• Multi-Game Decision Transformers
• A Generalist Agent
• A Model Selection Approach for Corruption Robust Reinforcement Learning
• Robust Stochastic Linear Contextual Bandits Under Adversarial Attacks
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Improved Corruption Robust Algorithms for Episodic Reinforcement\n Learning
• Achieving Near Instance-Optimality and Minimax-Optimality in Stochastic and Adversarial Linear Bandits Simultaneously
• Robust Policy Gradient against Strong Data Corruption
• Language Models are Few-Shot Learners
• Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning
• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
• Adaptive Reward-Poisoning Attacks against Reinforcement Learning
• Dynamic Backdoor Attacks Against Machine Learning Models
• Corruption-robust exploration in episodic reinforcement learning
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Policy Poisoning in Batch Reinforcement Learning and Control
• What You See May Not Be What You Get: UCB Bandit Algorithms Robust to ε-Contamination
• Attacking Vision-based Perception in End-to-End Autonomous Driving Models
• CopyCAT: Taking Control of Neural Policies with Constant Attacks
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Stochastic bandits robust to adversarial corruptions
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Robust Deep Reinforcement Learning with Adversarial Attacks
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• Proximal Policy Optimization Algorithms
• Attention Is All You Need
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Tactics of Adversarial Attack on Deep Reinforcement Learning Agents
• Adversarial Attacks on Neural Network Policies
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• Data Poisoning Attacks on Factorization-Based Collaborative Filtering
• Practical Black-Box Attacks against Machine Learning
• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks

Related

• Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learning
• Adaptive Reward-Poisoning Attacks against Reinforcement Learning
• Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments
• Online Poisoning Attack Against Reinforcement Learning under Black-box Environments
• Disturbing Reinforcement Learning Agents with Corrupted Rewards