Should You Use Your Large Language Model to Explore or Exploit?

2025/01/31 by Harris, Keegan, Slivkins, Aleksandrs · 3 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2502.00225

Abstract

We evaluate the ability of the current generation of large language models (LLMs) to help a decision-making agent facing an exploration-exploitation tradeoff. We use LLMs to explore and exploit in silos in various (contextual) bandit tasks. We find that while the current LLMs often struggle to exploit, in-context mitigations may be used to substantially improve performance for small-scale tasks. However even then, LLMs perform worse than a simple linear regression. On the other hand, we find that LLMs do help at exploring large action spaces with inherent semantics, by suggesting suitable candidates to explore.

Cited by

• Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning
• Comparing Exploration-Exploitation Strategies of LLMs and Humans: Insights from Standard Multi-armed Bandit Experiments
• Exploration and Exploitation Errors Are Measurable for Language Model Agents

Related

• Can large language models explore in-context?
• EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration
• Do Large Language Models Know What They Are Capable Of?
• Eight Things to Know about Large Language Models
• A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Bandits