2025/11/19 by Pengcheng Dai, Dai, Pengcheng, He Wang +5
Computer Science · Engineering · #Adaptive Dynamic Programming Control #Reinforcement Learning in Robotics #Smart Grid Security and Resilience #cs.MA
paper · pdf · doi:10.48550/arxiv.2511.15053
openalex publication_date 2025/11/19 · openalex created_date 2025/11/23 · openalex updated_date 2026/07/28
This paper investigates constrained multi-agent reinforcement learning (CMARL) in coupled environments, where agents collaboratively maximize the sum of local objectives while satisfying individual safety constraints. Existing studies face two limitations: (1) most rely on independent policies that fail to capture complex interactions in coupled environments; and (2) agents require the global Lagrange multipliers, which are sensitive learned variables whose global sharing risks exposing private agent-specific information. To overcome these issues, we propose a framework where agents adopt coupled policies that depend on both the states and policy parameters of their κp-hop neighbors, where κp>0 denotes the coupling distance, and develop a distributed and scalable primal-dual (DSPD) algorithm wherein each agent accesses only information within a prescribed local neighborhood. In the proposed algorithm, agents exchange sensitive parameters only with immediate neighbors over a separate time-varying network, while maintaining local estimates to execute the coupled policy. We establish that the proposed algorithm achieves ε-first-order stationary convergence with approximation error O(γ^\fracκ+1κp), where κ>0 is the truncated distance and γ∈(0,1) is discount factor. Simulations on a wireless access-control network demonstrate that the proposed algorithm outperforms existing state-of-the-art algorithms, validating its effectiveness.