Gradient Projection onto Historical Descent Directions for Communication-Efficient Federated Learning

2025/11/05 by Arnaud Descours, Descours, Arnaud, Léonard Deroose +3
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Optimization and Control (math.OC) #Privacy-Preserving Technologies in Data #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2511.05593

openalex publication_date 2025/11/05 · openalex created_date 2025/11/06 · openalex updated_date 2026/07/28

Abstract

Federated Learning (FL) enables decentralized model training across multiple clients while optionally preserving data privacy. However, communication efficiency remains a critical bottleneck, particularly for large-scale models. In this work, we introduce two complementary algorithms: ProjFL, designed for unbiased compressors, and ProjFL+EF, tailored for biased compressors through an Error Feedback mechanism. Both methods rely on projecting local gradients onto a shared client-server subspace spanned by historical descent directions, enabling efficient information exchange with minimal communication overhead. We establish convergence guarantees for both algorithms under strongly convex, convex, and non-convex settings. Empirical evaluations on standard FL classification benchmarks with deep neural networks show that ProjFL and ProjFL+EF achieve accuracy comparable to existing baselines while substantially reducing communication costs.

Citations

• Compressed and distributed least-squares regression: convergence rates with applications to Federated Learning
• Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?
• Momentum Provably Improves Error Feedback!
• Adaptive Compression for Communication-Efficient Distributed Training
• EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedback
• FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learning
• EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback
• MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization
• Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
• Advances and Open Problems in Federated Learning
• A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and\n Coordinate Descent
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Distributed Learning with Compressed Gradient Differences
• The Convergence of Sparsified Gradient Methods
• Sparsified SGD with Memory
• Distributed learning with compressed gradients
• Deep Residual Learning for Image Recognition
• 8-Bit Approximations for Parallelism in Deep Learning

Related

• CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning
• E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing
• Communication-Efficient Federated Learning with Accelerated Client Gradient
• Adaptive Control of Client Selection and Gradient Compression for Efficient Federated Learning
• Decentralized Federated Learning: Balancing Communication and Computing Costs