Generalization in Reinforcement Learning with Selective Noise Injection\n and Information Bottleneck

2019/10/28 by Maximilian Igl, Kamil Ciosek, Igl, Maximilian +11 · 7 citations
Computer Science · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #Data Stream Mining Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1910.12911

openalex publication_date 2019/10/28 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28

Abstract

The ability for policies to generalize to new environments is key to the\nbroad application of RL agents. A promising approach to prevent an agent's\npolicy from overfitting to a limited set of training environments is to apply\nregularization techniques originally developed for supervised learning.\nHowever, there are stark differences between supervised learning and RL. We\ndiscuss those differences and propose modifications to existing regularization\ntechniques in order to better adapt them to RL. In particular, we focus on\nregularization techniques relying on the injection of noise into the learned\nfunction, a family that includes some of the most widely used approaches such\nas Dropout and Batch Normalization. To adapt them to RL, we propose Selective\nNoise Injection (SNI), which maintains the regularizing effect the injected\nnoise has, while mitigating the adverse effects it has on the gradient quality.\nFurthermore, we demonstrate that the Information Bottleneck (IB) is a\nparticularly well suited regularization technique for RL as it is effective in\nthe low-data regime encountered early on in training RL agents. Combining the\nIB with SNI, we significantly outperform current state of the art results,\nincluding on the recently proposed generalization benchmark Coinrun.\n

Cited by

• Bias-Variance Trade-off for Clipped Stochastic First-Order Methods: From Bounded Variance to Infinite Mean
• Leveraging weights signals -- Predicting and improving generalizability in reinforcement learning
• NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
• Instance based Generalization in Reinforcement Learning
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration
• Active Reinforcement Learning over MDPs

Related

• Quantifying Generalization in Reinforcement Learning
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning
• Improving Generalization in Reinforcement Learning with Mixture Regularization
• A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
• DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization