Strategic Coordination for Evolving Multi-agent Systems: A Hierarchical Reinforcement and Collective Learning Approach

2025/09/22 by Chuhao Qin, Evangelos Pournaras, Qin, Chuhao +1
Computer Science · Engineering · #Advanced Research in Systems and Signal Processing #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multi-Agent Systems and Negotiation #Multiagent Systems (cs.MA) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2509.18088

openalex publication_date 2025/09/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Decentralized combinatorial optimization in evolving multi-agent systems poses significant challenges, requiring agents to balance long-term decision-making, short-term optimized collective outcomes, while preserving autonomy of interactive agents under unanticipated changes. Reinforcement learning offers a way to model sequential decision-making through dynamic programming to anticipate future environmental changes. However, applying multi-agent reinforcement learning (MARL) to decentralized combinatorial optimization problems remains an open challenge due to the exponential growth of the joint state-action space, high communication overhead, and privacy concerns in centralized training. To address these limitations, this paper proposes Hierarchical Reinforcement and Collective Learning (HRCL), a novel approach that leverages both MARL and decentralized collective learning based on a hierarchical framework. Agents take high-level strategies using MARL to group possible plans for action space reduction and constrain the agent behavior for Pareto optimality. Meanwhile, the low-level collective learning layer ensures efficient and decentralized coordinated decisions among agents with minimal communication. Extensive experiments in a synthetic scenario and real-world smart city application models, including energy self-management and drone swarm sensing, demonstrate that HRCL significantly improves performance, scalability, and adaptability compared to the standalone MARL and collective learning approaches, achieving a win-win synthesis solution.

Citations

• Coordination of Drones at Scale: Decentralized Energy-aware Swarm Intelligence for Spatio-temporal Sensing
• 3, 2, 1, Drones Go! A Testbed to Take off UAV Swarm Intelligence for Distributed Sensing
• Reinforcement Learning from Partial Observation: Linear Function Approximation with Provable Sample Efficiency
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism
• Exact Reduction of Huge Action Spaces in General Reinforcement Learning
• Combining Reinforcement Learning and Constraint Programming for Combinatorial Optimization
• Action Space Shaping in Deep Reinforcement Learning
• Reinforcement Learning for Combinatorial Optimization: A Survey
• Macro action selection with deep reinforcement learning in StarCraft
• Optimized Path Planning for Inspection by Unmanned Aerial Vehicles Swarm with Energy Constraints
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• The Complexity of Decentralized Control of Markov Decision Processes
• The Complexity of Decentralized Control of Markov Decision Processes

Related

• Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advances
• Moving Forward in Formation: A Decentralized Hierarchical Learning Approach to Multi-Agent Moving Together
• Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning
• General Principles of Learning-Based Multi-Agent Systems
• Learning to Lead Themselves: Agentic AI in MAS using MARL