Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices

2021/11/03 by Junliang Lin, Jun-Liang Lin, Lin, Jun-Liang +3
Computer Science · #Advanced Neural Network Applications #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Parallel #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC) #cs.DC #cs.LG

paper · pdf · doi:10.48550/arxiv.2111.02489

arxiv created 2021/11/03 · openalex publication_date 2021/11/03 · arxiv updated 2021/11/05 · openalex created_date 2021/11/08 · openalex updated_date 2026/07/28

Abstract

The inference of Neural Networks is usually restricted by the resources (e.g., computing power, memory, bandwidth) on edge devices. In addition to improving the hardware design and deploying efficient models, it is possible to aggregate the computing power of many devices to enable the machine learning models. In this paper, we proposed a novel method of exploiting model parallelism to separate a neural network for distributed inferences. To achieve a better balance between communication latency, computation latency, and performance, we adopt neural architecture search (NAS) to search for the best transmission policy and reduce the amount of communication. The best model we found decreases by 86.6% of the amount of data transmission compared to the baseline and does not impact performance much. Under proper specifications of devices and configurations of models, our experiments show that the inference of large neural networks on edge clusters can be distributed and accelerated, which provides a new solution for the deployment of intelligent applications in the internet of things (IoT).

Citations

• ImageNet classification with deep convolutional neural networks
• Neural Architecture Search with Reinforcement Learning
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• AutoAugment: Learning Augmentation Policies from Data
• Self-Training With Noisy Student Improves ImageNet Classification

Related

• Design and Prototyping Distributed CNN Inference Acceleration in Edge Computing
• Partitioning and Placement of Deep Neural Networks on Distributed Edge Devices to Maximize Inference Throughput
• DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices
• Communication-Computation Trade-Off in Resource-Constrained Edge Inference
• Neural Architecture Search for Improving Latency-Accuracy Trade-off in Split Computing