Evolution Strategies as a Scalable Alternative to Reinforcement Learning

2017/03/10 by Tim Salimans, Salimans, Tim, Jonathan Ho +7 · 1 voice · 102 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Metaheuristic Optimization Algorithms Research #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics #cs.AI #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1703.03864

openalex publication_date 2017/03/10 · arxiv published 2017/03/10 · arxiv updated 2017/09/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We explore the use of Evolution Strategies (ES), a class of black box optimization algorithms, as an alternative to popular MDP-based RL techniques such as Q-learning and Policy Gradients. Experiments on MuJoCo and Atari show that ES is a viable solution strategy that scales extremely well with the number of CPUs available: By using a novel communication strategy based on common random numbers, our ES implementation only needs to communicate scalars, making it possible to scale to over a thousand parallel workers. This allows us to solve 3D humanoid walking in 10 minutes and obtain competitive results on most Atari games after one hour of training. In addition, we highlight several advantages of ES as a black box optimization technique: it is invariant to action frequency and delayed rewards, tolerant of extremely long horizons, and does not need temporal discounting or value function approximation.

Citations

• Improved Techniques for Training GANs
• Episodic Exploration for Deep Deterministic Policies: An Application to\n StarCraft Micromanagement Tasks
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning

Cited by

• RIS-Enabled Smart Wireless Environments: Fundamentals and Distributed Optimization
• CosmoCore-Evo: Evolutionary Dream-Replay Reinforcement Learning for Adaptive Code Generation
• NOVA: Discovering Well-Conditioned Winograd Transforms through Numerical Optimization of Vandermonde Arithmetic
• Hypernetworks That Evolve Themselves
• BEP: A Binary Error Propagation Algorithm for Binary Neural Networks Training
• Evolution Strategies at the Hyperscale
• Deep active inference
• Harnessing Bounded-Support Evolution Strategies for Policy Refinement
• SPIDER: Scalable Physics-Informed Dexterous Retargeting
• PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explore
• Periodic Skill Discovery
• ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Models
• Organismal Agency and Rapid Adaptation: The Phenopoiesis Algorithm for Phenotype-First Evolution
• Feature Matching-Based Gait Phase Prediction for Obstacle Crossing Control of Powered Transfemoral Prosthesis
• Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineering
• Dopamine-driven synaptic credit assignment in neural networks
• HGraphScale: Hierarchical Graph Learning for Autoscaling Microservice Applications in Container-based Cloud Computing
• A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation
• The Emergence of Complex Behavior in Large-Scale Ecological Environments
• Lark: Biologically Inspired Neuroevolution for Multi-Stakeholder LLM Agents
• Patch-wise Attack for Fooling Deep Neural Network
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• EA4LLM: A Gradient-Free Approach to Large Language Model Optimization via Evolutionary Algorithms
• Switching Isotropic and Directional Exploration with Parameter Space\n Noise in Deep Reinforcement Learning
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• Evolved Policy Gradients
• A Stochastic Derivative Free Optimization Method with Momentum
• Searching in the Forest for Local Bayesian Optimization
• Tasks, stability, architecture, and compute: Training more effective\n learned optimizers, and using them to train themselves
• A Multi-Agent Framework for Stateful Inference-Time Search
• GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning
• Distributionally Constrained Black-Box Stochastic Gradient Estimation and Optimization
• Zeroth-Order Algorithms for Nonconvex Minimax Problems with Improved Complexities
• On Discovering Algorithms for Adversarial Imitation Learning
• TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks
• Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedback
• Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
• Continuous-Discrete Reinforcement Learning for Hybrid Control in\n Robotics
• Adaptive Sampling Quasi-Newton Methods for Derivative-Free Stochastic\n Optimization
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks
• Joint Active RIS Configuration and User Power Control for Localization: A Neuroevolution-Based Approach
• Population-Based Evolution Optimizes a Meta-Learning Objective
• Variance Reduction for Evolution Strategies via Structured Control Variates
• AdaDGS: An adaptive black-box optimization method with a nonlocal directional Gaussian smoothing gradient
• Generalization Guarantees for Imitation Learning
• Neural Rate Control for Video Encoding using Imitation Learning
• Learning Index Selection with Structured Action Spaces
• On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator
• Ranking Cost: Building An Efficient and Scalable Circuit Routing Planner with Evolution-Based Optimization
• Mirror Natural Evolution Strategies
• Decoupling Search and Learning in Neural Net Training
• Zeroth-Order Supervised Policy Improvement
• Learning Self-Imitating Diverse Policies
• Greedy Layerwise Learning Can Scale to ImageNet
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• MORSE: Multi-Objective Reinforcement Learning via Strategy Evolution for Supply Chain Optimization
• CHALET: Cornell House Agent Learning Environment
• Analyzing Reinforcement Learning Benchmarks with Random Weight Guessing
• HyperNCA: Growing Developmental Networks with Neural Cellular Automata
• Learning Convex Optimization Control Policies
• The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
• GPLight+: A Genetic Programming Method for Learning Symmetric Traffic Signal Control Policy
• Distilling Policy Distillation
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Ray: A Distributed Framework for Emerging AI Applications
• HEAS: Hierarchical Evolutionary Agent Simulation Framework for Cross-Scale Modeling and Multi-Objective Search
• Breaking Through Barren Plateaus: Reinforcement Learning Initializations for Deep Variational Quantum Circuits
• Momentum Accelerates Evolutionary Dynamics
• Efficient Environment Design for Multi-Robot Navigation via Continuous Control
• Reward Tweaking: Maximizing the Total Reward While Planning for Short Horizons
• Optimizing Sponsored Search Ranking Strategy by Deep Reinforcement Learning
• Reinforcement learning in densely recurrent biological networks
• Adaptive Wind Driven Optimization Trained Artificial Neural Networks
• Playing Atari Space Invaders with Sparse Cosine Optimized Policy Evolution
• Meta Reinforcement Learning with Distribution of Exploration Parameters Learned by Evolution Strategies
• \(X\)-evolve: Solution space evolution powered by large language models
• Modelling zebrafish collective behaviours with multilayer perceptrons optimised by evolutionary algorithms
• Winning Isn't Everything: Enhancing Game Development with Intelligent Agents
• Tuna: A Static Analysis Approach to Optimizing Deep Neural Networks
• Evolutionary Generative Optimization: Towards Fully Data-Driven Evolutionary Optimization via Generative Learning
• Importance mixing: Improving sample reuse in evolutionary policy search methods
• A Frank-Wolfe Framework for Efficient and Effective Adversarial Attacks
• Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversity
• Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
• SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
• Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model
• CoNES: Convex Natural Evolutionary Strategies
• Testing the Genomic Bottleneck Hypothesis in Hebbian Meta-Learning
• Learning Algorithmic Solutions to Symbolic Planning Tasks with a Neural Computer Architecture
• Sparse Stochastic Zeroth-Order Optimization with an Application to\n Bandit Structured Prediction
• The Price equation reveals a universal force-metric-bias law of algorithmic learning and natural selection
• Evolution Strategies Converges to Finite Differences
• A Hybrid Gradient Method to Designing Bayesian Experiments for Implicit Models
• Self-Evolutionary Optimization for Pareto Front Learning
• Improving Evolutionary Strategies with Generative Neural Networks
• Limited-Memory Matrix Adaptation for Large Scale Black-box Optimization
• Recruitment-imitation Mechanism for Evolutionary Reinforcement Learning
• Network of Evolvable Neural Units: Evolving to Learn at a Synaptic Level
• Proximal Policy Optimization with Mixed Distributed Training
• Recurrent Control Nets for Deep Reinforcement Learning

Discussions

• Evolution Strategies as a Scalable Alternative to Reinforcement Learning [hn, 3 points, 0 comments]

Related

• Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari
• Shaped Policy Search for Evolutionary Strategies using Waypoints
• ES Is More Than Just a Traditional Finite-Difference Approximator
• Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey
• Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks