γ-FedHT: Stepsize-Aware Hard-Threshold Gradient Compression in Federated Learning

2025/05/18 by Rongwei Lu, Yutong Jiang, Lu, Rongwei +11 · 1 citation
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.12479

openalex publication_date 2025/05/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Gradient compression can effectively alleviate communication bottlenecks in Federated Learning (FL). Contemporary state-of-the-art sparse compressors, such as Top-k, exhibit high computational complexity, up to O(dlog₂k), where d is the number of model parameters. The hard-threshold compressor, which simply transmits elements with absolute values higher than a fixed threshold, is thus proposed to reduce the complexity to O(d). However, the hard-threshold compression causes accuracy degradation in FL, where the datasets are non-IID and the stepsize γ is decreasing for model convergence. The decaying stepsize reduces the updates and causes the compression ratio of the hard-threshold compression to drop rapidly to an aggressive ratio. At or below this ratio, the model accuracy has been observed to degrade severely. To address this, we propose γ-FedHT, a stepsize-aware low-cost compressor with Error-Feedback to guarantee convergence. Given that the traditional theoretical framework of FL does not consider Error-Feedback, we introduce the fundamental conversation of Error-Feedback. We prove that γ-FedHT has the convergence rate of O((1)/(T)) (T representing total training iterations) under μ-strongly convex cases and O((1)/(√(T))) under non-convex cases, same as FedAVG. Extensive experiments demonstrate that γ-FedHT improves accuracy by up to 7.42% over Top-k under equal communication traffic on various non-IID image datasets.

Citations

• EControl: Fast Distributed Optimization with Compression and Error Control
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges
• DoCoFL: Downlink Compression for Cross-Device Federated Learning
• Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning
• Optimal Rate Adaption in Federated Learning with Compressed Communications
• Rethinking gradient sparsification as total error minimization
• CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning
• EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback
• Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates
• Federated Learning on Non-IID Data Silos: An Experimental Study
• FEDZIP: A Compression Framework for Communication-Efficient Federated Learning
• Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
• On Communication Compression for Distributed Optimization on Heterogeneous Data
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• Federated Learning With Quantized Global Model Updates
• Advances and Open Problems in Federated Learning
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• On the Convergence of FedAvg on Non-IID Data
• Robust and Communication-Efficient Federated Learning from Non-IID Data
• The Convergence of Sparsified Gradient Methods
• Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
• Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Pointer Sentinel Mixture Models

Cited by

• Per-element Secure Aggregation against Data Reconstruction Attacks in Federated Learning

Related

• E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing
• CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning
• Gradient Projection onto Historical Descent Directions for Communication-Efficient Federated Learning
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis