Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning

2025/02/10 by Lorenc, Matyáš, Neruda, Roman · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE)

paper · doi:10.48550/arxiv.2502.06301

Abstract

In this paper, we experiment with novelty-based variants of OpenAI-ES, the NS-ES and NSR-ES algorithms, and evaluate their effectiveness in training complex, transformer-based architectures designed for the problem of reinforcement learning, such as Decision Transformers. We also test if we can accelerate the novelty-based training of these larger models by seeding the training with a pretrained models. The experimental results were mixed. NS-ES showed progress, but it would clearly need many more iterations for it to yield interesting agents. NSR-ES, on the other hand, proved quite capable of being straightforwardly used on larger models, since its performance appears as similar between the feed-forward model and Decision Transformer, as it was for the OpenAI-ES in our previous work.

Cited by

• EA4LLM: A Gradient-Free Approach to Large Language Model Optimization via Evolutionary Algorithms

Related

• Utilizing Evolution Strategies to Train Transformers in Reinforcement Learning
• Transformers in Reinforcement Learning: A Survey
• A Survey on Transformers in Reinforcement Learning
• Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining
• RL + Transformer = A General-Purpose Problem Solver