Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries

2024/03/15 by Swetha Ganesh, Ganesh, Swetha, Jiayu Chen +5 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Matrix Theory and Algorithms #Optimization and Control (math.OC) #Optimization and Search Problems #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2403.09940

openalex publication_date 2024/03/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Federated Reinforcement Learning (FRL) allows multiple agents to collaboratively build a decision making policy without sharing raw trajectories. However, if a small fraction of these agents are adversarial, it can lead to catastrophic results. We propose a policy gradient based approach that is robust to adversarial agents which can send arbitrary values to the server. Under this setting, our results form the first global convergence guarantees with general parametrization. These results demonstrate resilience with adversaries, while achieving optimal sample complexity of order O( (1)/(Nε²) ( 1+ (f²)/(N))), where N is the total number of agents and f

Cited by

• Parameter-Free Federated TD Learning with Markov Noise in Heterogeneous Environments
• Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learning

Related

• On Global Convergence Rates for Federated Softmax Policy Gradient under Heterogeneous Environments
• Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee
• The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communication
• Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updates
• Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning