Reinforced Language Models for Sequential Decision Making

2025/08/14 by Dilkes, Jim, Yazdanpanah, Vahid, Stein, Sebastian
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #I.2.8 #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.10839

Abstract

Large Language Models (LLMs) show potential as sequential decision-making agents, but their application is often limited due to a reliance on large, computationally expensive models. This creates a need to improve smaller models, yet existing post-training methods are designed for single-turn interactions and cannot handle credit assignment in multi-step agentic tasks. To address this, we introduce Multi-Step Group-Relative Policy Optimization (MS-GRPO), a new algorithm for post-training LLM agents, grounded in formal Text-Mediated Stochastic Game (TSMG) and Language-Agent Policy (LAP) frameworks. For credit assignment, MS-GRPO attributes the entire cumulative episode reward to each individual episode step. We supplement this algorithm with a novel absolute-advantage-weighted episode sampling strategy that we show improves training performance. We evaluate our approach by post-training a 3-billion parameter model on Snake and Frozen Lake. Our experiments demonstrate that the method is effective in improving decision-making performance: our post-trained 3B parameter model outperforms a 72B parameter baseline by 50% on the Frozen Lake task. This work demonstrates that targeted post-training is a practical and efficient alternative to relying on model scale for creating sequential decision-making agents using LLMs.

Citations

• Reinforcement Learning for Reasoning in Large Language Models with One Training Example
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
• Qwen2.5 Technical Report
• AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
• ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning
• DPO Meets PPO: Reinforced Token Optimization for RLHF
• Reflect-RL: Two-Player Online RL Fine-Tuning for LMs
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Evaluation of Large Language Models for Decision Making in Autonomous Driving
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Reflexion: Language Agents with Verbal Reinforcement Learning
• ReAct: Synergizing Reasoning and Acting in Language Models
• Training language models to follow instructions with human feedback
• LoRA: Low-Rank Adaptation of Large Language Models
• Fine-Tuning Language Models from Human Preferences
• Attention, Learn to Solve Routing Problems!
• Proximal Policy Optimization Algorithms
• Prioritized Experience Replay
• End-to-End Training of Deep Visuomotor Policies
• Playing Atari with Deep Reinforcement Learning
• Reinforcement Learning: An Introduction

Related

• ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspective
• On the Modeling Capabilities of Large Language Models for Sequential Decision Making
• Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning
• Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design
• Training-Free Group Relative Policy Optimization