MasHost Builds It All: Autonomous Multi-Agent System Directed by Reinforcement Learning

2025/06/10 by Kuo‐Hsin Yang, Yang, Kuo, Linhui Yu +12
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Big Data and Digital Economy #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multiagent Systems (cs.MA) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2506.08507

openalex publication_date 2025/06/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Model (LLM)-driven Multi-agent systems (Mas) have recently emerged as a powerful paradigm for tackling complex real-world tasks. However, existing Mas construction methods typically rely on manually crafted interaction mechanisms or heuristic rules, introducing human biases and constraining the autonomous ability. Even with recent advances in adaptive Mas construction, existing systems largely remain within the paradigm of semi-autonomous patterns. In this work, we propose MasHost, a Reinforcement Learning (RL)-based framework for autonomous and query-adaptive Mas design. By formulating Mas construction as a graph search problem, our proposed MasHost jointly samples agent roles and their interactions through a unified probabilistic sampling mechanism. Beyond the accuracy and efficiency objectives pursued in prior works, we introduce component rationality as an additional and novel design principle in Mas. To achieve this multi-objective optimization, we propose Hierarchical Relative Policy Optimization (HRPO), a novel RL strategy that collaboratively integrates group-relative advantages and action-wise rewards. To our knowledge, our proposed MasHost is the first RL-driven framework for autonomous Mas graph construction. Extensive experiments on six benchmarks demonstrate that MasHost consistently outperforms most competitive baselines, validating its effectiveness, efficiency, and structure rationality.

Citations

• Large Language Model Agent: A Survey on Methodology, Applications and Challenges
• MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems
• MasRouter: Learning to Route LLMs for Multi-Agent Systems
• Multi-agent Architecture Search via Agentic Supernet
• Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning
• CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models
• Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning
• G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks
• AFlow: Automating Agentic Workflow Generation
• Automated Design of Agentic Systems
• LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions
• Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimization
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• TravelPlanner: A Benchmark for Real-World Planning with Language Agents
• Large Language Model based Multi-Agents: A Survey of Progress and Challenges
• Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering
• Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security
• Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding
• Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• AutoAgents: A Framework for Automatic Agent Generation
• AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
• TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPT
• Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration
• Large Language Models
• Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• Reflexion: Language Agents with Verbal Reinforcement Learning
• GPT-4 Technical Report
• LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Training Verifiers to Solve Math Word Problems
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• Measuring Mathematical Problem Solving With the MATH Dataset
• Measuring Massive Multitask Language Understanding
• Proximal Policy Optimization Algorithms
• Deep Reinforcement Learning: An Overview
• A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration

Related

• Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs
• Heterogeneous Group-Based Reinforcement Learning for LLM-based Multi-Agent Systems
• ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Tasks
• LLM Collaboration With Multi-Agent Reinforcement Learning
• Enhancing Multi-Agent Systems via Reinforcement Learning with LLM-based Planner and Graph-based Policy