Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithm

2024/06/10 by Ahmed Elbakary, Chaouki Ben Issaid, Elbakary, Ahmed +9 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Cooperative Communication and Network Coding #Cryptography and Data Security #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #Privacy-Preserving Technologies in Data #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2406.06655

openalex publication_date 2024/06/10 · openalex created_date 2024/06/13 · openalex updated_date 2026/07/28

Abstract

Federated learning is a machine learning approach where multiple devices collaboratively learn with the help of a parameter server by sharing only their local updates. While gradient-based optimization techniques are widely adopted in this domain, the curvature information that second-order methods exhibit is crucial to guide and speed up the convergence. This paper introduces a scalable second-order method, allowing the adoption of curvature information in federated large models. Our method, coined Fed-Sophia, combines a weighted moving average of the gradient with a clipping operation to find the descent direction. In addition to that, a lightweight estimation of the Hessian's diagonal is used to incorporate the curvature information. Numerical evaluation shows the superiority, robustness, and scalability of the proposed Fed-Sophia scheme compared to first and second-order baselines.

Cited by

• Accelerated Training of Federated Learning via Second-Order Methods

Related

• Accelerated Training of Federated Learning via Second-Order Methods
• FedOSAA: Improving Federated Learning with One-Step Anderson Acceleration
• FedSSO: A Federated Server-Side Second-Order Optimization Algorithm
• FLECS: A Federated Learning Second-Order Framework via Compression and Sketching
• On Second-order Optimization Methods for Federated Learning