Making Reinforcement Learning Work on Swimmer

2022/08/16 by Maël Franceschetti, Franceschetti, Maël, Coline Lacoux +5 · 1 citation
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2208.07587

openalex publication_date 2022/08/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The SWIMMER environment is a standard benchmark in reinforcement learning (RL). In particular, it is often used in papers comparing or combining RL methods with direct policy search methods such as genetic algorithms or evolution strategies. A lot of these papers report poor performance on SWIMMER from RL methods and much better performance from direct policy search methods. In this technical report we show that the low performance of RL methods on SWIMMER simply comes from the inadequate tuning of an important hyper-parameter, the discount factor. Furthermore we show that, by setting this hyper-parameter to a correct value, the issue can be easily fixed. Finally, for a set of often used RL algorithms, we provide a set of successful hyper-parameters obtained with the Stable Baselines3 library and its RL Zoo.

Cited by

• SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling

Related

• Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning
• Performance Optimization of Ratings-Based Reinforcement Learning
• One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF
• Evaluating Parameter Efficient Methods for RLVR
• Hyperparameters in Reinforcement Learning and How To Tune Them