Disentangling Exploration of Large Language Models by Optimal Exploitation

2025/01/15 by Tim Grams, Grams, Tim, Patrick Betz +4
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2501.08925

openalex publication_date 2025/01/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Exploration is a crucial skill for in-context reinforcement learning in unknown environments. However, it remains unclear if large language models can effectively explore a partially hidden state space. This work isolates exploration as the sole objective, tasking an agent with gathering information that enhances future returns. Within this framework, we argue that measuring agent returns is not sufficient for a fair evaluation. Hence, we decompose missing rewards into their exploration and exploitation components based on the optimal achievable return. Experiments with various models reveal that most struggle to explore the state space, and weak exploration is insufficient. Nevertheless, we found a positive correlation between exploration performance and reasoning capabilities. Our decomposition can provide insights into differences in behaviors driven by prompt engineering, offering a valuable tool for refining performance in exploratory tasks.

Cited by

• Survey: Multi-Armed Bandits Meet Large Language Models

Related

• EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration
• Can large language models explore in-context?
• Toward Efficient Exploration by Large Language Model Agents
• CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
• Guiding Pretraining in Reinforcement Learning with Large Language Models