On the Communication Latency of Wireless Decentralized Learning

2020/02/10 by Navid Naderializadeh, Naderializadeh, Navid
Computer Science · Engineering · #Cooperative Communication and Network Coding #FOS: Computer and information sciences #Indoor and Outdoor Localization Technologies #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Wireless Networks and Protocols

paper · pdf · doi:10.48550/arxiv.2002.04069

openalex publication_date 2020/02/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider a wireless network comprising n nodes located within a circular area of radius R, which are participating in a decentralized learning algorithm to optimize a global objective function using their local datasets. To enable gradient exchanges across the network, we assume each node communicates only with a set of neighboring nodes, which are within a distance R n^-β of itself, where β∈(0,(1)/(2)). We use tools from network information theory and random geometric graph theory to show that the communication delay for a single round of exchanging gradients on all the links throughout the network scales as O(\fracn^2-3ββlog n), increasing (at different rates) with both the number of nodes and the gradient exchange threshold distance.

Citations

• Sequence to Sequence Learning with Neural Networks
• ImageNet classification with deep convolutional neural networks
• Decentralized Stochastic Optimization and Gossip Algorithms with\n Compressed Communication
• LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning

Related

• Network-Density-Controlled Decentralized Parallel Stochastic Gradient Descent in Wireless Systems
• Decentralized Learning over Wireless Networks with Broadcast-Based Subgraph Sampling
• Communication-Efficient and Distributed Learning Over Wireless Networks:\n Principles and Applications
• Decentralized Learning over Wireless Networks: The Effect of Broadcast with Random Access
• Communication-efficient Decentralized Local SGD over Undirected Networks