Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs

2025/10/13 by Yujie Zhao, Lanxiang Hu, Zhao, Yujie +11
Computer Science · Decision Sciences · #Auction Theory and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multiagent Systems (cs.MA) #Open Source Software Innovations

paper · pdf · doi:10.48550/arxiv.2510.11062

openalex publication_date 2025/10/13 · openalex created_date 2025/10/17 · openalex updated_date 2026/08/01

Abstract

Multi-agent systems (MAS) and reinforcement learning (RL) are widely used to enhance the agentic capabilities of large language models (LLMs). MAS improves task performance through role-based orchestration, while RL uses environmental rewards to learn stronger policies, such as GRPO-style optimization. However, applying on-policy RL to MAS remains underexplored and presents unique challenges. Algorithmically, standard GRPO grouping assumptions break down because prompts vary by role and by turn. System-wise, the training stack must support MAS-workflow rollouts and on-policy updates for both single-policy and multi-policy models. We propose AT-GRPO, which includes (i) an agent- and turn-wise grouped RL algorithm tailored to MAS and (ii) a training system that supports both single- and multi-policy regimes. Across game, planning, coding, and math tasks, AT-GRPO delivers substantial gains. On long-horizon planning, it increases accuracy from a 14.0 to 47.0 percent single-agent RL baseline to 96.0 to 99.5 percent. It also improves reasoning performance, with average gains of 3.87 to 7.62 percent on coding tasks and 9.0 to 17.93 percent on math. Code and environments are available at: https://github.com/pettingllms-ai/PettingLLMs.

Citations

• SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
• Heterogeneous Group-Based Reinforcement Learning for LLM-based Multi-Agent Systems
• Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning
• X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMs
• lmgame-Bench: How Good are LLMs at Playing Games?
• Group-in-Group Policy Optimization for LLM Agent Training
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• MARFT: Multi-Agent Reinforcement Fine-Tuning
• ToolRL: Reward is All Tool Learning Needs
• Why Do Multi-Agent LLM Systems Fail?
• MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning
• CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance
• OrcaLoca: An LLM Agent Framework for Software Issue Localization
• SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling
• DrugAgent: Automating AI-aided Drug Discovery Programming through LLM Multi-Agent Collaboration
• Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning
• Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence
• Mixture-of-Agents Enhances Large Language Model Capabilities
• MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Large Language Model based Multi-Agents: A Survey of Progress and Challenges
• The Rise and Potential of Large Language Model Based Agents: A Survey
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Toolformer: Language Models Can Teach Themselves to Use Tools
• ReAct: Synergizing Reasoning and Acting in Language Models
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• Training Verifiers to Solve Math Word Problems
• GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
• Multi-Agent Reinforcement Learning for Problems with Combined Individual and Team Reward
• Reward Design in Cooperative Multi-agent Reinforcement Learning for Packet Routing
• AI Benchmark Half-Life in Recursive Corpora: A Theory of Validity Decay under Semantic Leakage and Regeneration
• OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
• Reinforcement Learning: An Introduction

Cited by

• Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation

Related

• LLM Collaboration With Multi-Agent Reinforcement Learning
• LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions
• MasHost Builds It All: Autonomous Multi-Agent System Directed by Reinforcement Learning
• Heterogeneous Group-Based Reinforcement Learning for LLM-based Multi-Agent Systems
• Multi-Agent Tool-Integrated Policy Optimization