Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL

2024/10/02 by Ghada Sokar, Sokar, Ghada, Johan Obando-Ceron +7 · 2 citations
Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2410.01930

openalex publication_date 2024/10/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The use of deep neural networks in reinforcement learning (RL) often suffers from performance degradation as model size increases. While soft mixtures of experts (SoftMoEs) have recently shown promise in mitigating this issue for online RL, the reasons behind their effectiveness remain largely unknown. In this work we provide an in-depth analysis identifying the key factors driving this performance gain. We discover the surprising result that tokenizing the encoder output, rather than the use of multiple experts, is what is behind the efficacy of SoftMoEs. Indeed, we demonstrate that even with an appropriately scaled single expert, we are able to maintain the performance gains, largely thanks to tokenization.

Cited by

• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents

Related

• Mixtures of Experts Unlock Parameter Scaling for Deep RL
• Mixture of Experts in a Mixture of RL settings
• Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning
• SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization