Integrating Large Language Models and Reinforcement Learning for Non-Linear Reasoning

2024/10/17 by Yoav Alon, Alon, Yoav, Cristina David +1
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multi-Agent Systems and Negotiation #Natural Language Processing Techniques #Programming Languages (cs.PL)

paper · pdf · doi:10.48550/arxiv.2410.13501

openalex publication_date 2024/10/17 · openalex created_date 2024/10/21 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) were shown to struggle with long-term planning, which may be caused by the limited way in which they explore the space of possible solutions. We propose an architecture where a Reinforcement Learning (RL) Agent guides an LLM's space exploration: (1) the Agent has access to domain-specific information, and can therefore make decisions about the quality of candidate solutions based on specific and relevant metrics, which were not explicitly considered by the LLM's training objective; (2) the LLM can focus on generating immediate next steps, without the need for long-term planning. We allow non-linear reasoning by exploring alternative paths and backtracking. We evaluate this architecture on the program equivalence task, and compare it against Chain of Thought (CoT) and Tree of Thoughts (ToT). We assess both the downstream task, denoting the binary classification, and the intermediate reasoning steps. Our approach compares positively against CoT and ToT.

Related

• Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought
• CoT-Space: A Theoretical Framework for Internal Slow-Thinking via Reinforcement Learning
• Rectifying LLM Thought from Lens of Optimization
• Training Language Models to Reason Efficiently
• A Survey of Reinforcement Learning for Large Reasoning Models