Communication-Efficient Neural Tangent Kernels for Heterogeneous Decentralized Federated Learning

2025/12/14 by Xia Li, Xia, Li
Computer Science · #68T05 #Advanced Neural Network Applications #Distributed #FOS: Computer and information sciences #I.2.11 #I.2.6 #Machine Learning (cs.LG) #Parallel #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2512.12737

openalex publication_date 2025/12/14 · openalex created_date 2025/12/17 · openalex updated_date 2026/07/28

Abstract

Decentralized federated learning (DFL) enables collaborative model training without a central server, but converges slowly under statistical heterogeneity. Recent work has shown that neural tangent kernel (NTK) methods achieve faster convergence than gradient-based updates in DFL, while momentum has proven effective for accelerating gradient-based FL. However, applying momentum to NTK updates can destabilize training under heterogeneous data. We propose SPARK, which addresses this instability with a stage-wise annealed soft-label regularizer evaluated on neighborhood-aggregated data, so that momentum can accelerate NTK updates stably. Under high heterogeneity, SPARK converges about 3× faster than baselines and lowers the total communication to a target accuracy by up to about 70%, and it attains higher accuracy across heterogeneity levels. We further study random projection as an optional Jacobian-compression strategy for bandwidth-constrained settings. We validate the approach across multiple datasets, network topologies, and heterogeneity levels.

Citations

• Non-Linear Trajectory Modeling for Multi-Step Gradient Inversion Attacks in Federated Learning
• NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernel
• Dynamic Heterogeneous Federated Learning with Multi-Level Prototypes
• A Survey on Decentralized Federated Learning
• Momentum Benefits Non-IID Federated Learning Simply and Provably
• Decentralized Federated Learning: A Survey and Perspective
• Surrogate Model Extension (SME): A Fast and Accurate Weight Update Attack on Federated Learning
• Improving the Model Consistency of Decentralized Federated Learning
• TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels
• DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training
• Neural Tangent Kernel Empowered Federated Learning
• Decentralized Federated Averaging
• Advances and Open Problems in Federated Learning
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Federated Optimization in Heterogeneous Networks
• LEAF: A Benchmark for Federated Settings
• Measuring the Intrinsic Dimension of Objective Landscapes
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Federated Learning: Strategies for Improving Communication Efficiency
• Deep Residual Learning for Image Recognition
• Gradient-based learning applied to document recognition

Related

• FLeNS: Federated Learning with Enhanced Nesterov-Newton Sketch
• Improving Federated Learning Communication Efficiency with Global Momentum Fusion for Gradient Compression Schemes
• Distributed Low-Communication Training with Decoupled Momentum Optimization
• CoDeC: Communication-Efficient Decentralized Continual Learning
• DIGEST: Fast and Communication Efficient Decentralized Learning with Local Updates