Communication-Efficient Split Learning via Adaptive Feature-Wise Compression

2023/07/20 by Yongjeong Oh, Oh, Yongjeong, Jaeho Lee +5 · 5 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Distributed #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Parallel #Remote-Sensing Image Classification #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2307.10805

openalex publication_date 2023/07/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper proposes a novel communication-efficient split learning (SL) framework, named SplitFC, which reduces the communication overhead required for transmitting intermediate feature and gradient vectors during the SL training process. The key idea of SplitFC is to leverage different dispersion degrees exhibited in the columns of the matrices. SplitFC incorporates two compression strategies: (i) adaptive feature-wise dropout and (ii) adaptive feature-wise quantization. In the first strategy, the intermediate feature vectors are dropped with adaptive dropout probabilities determined based on the standard deviation of these vectors. Then, by the chain rule, the intermediate gradient vectors associated with the dropped feature vectors are also dropped. In the second strategy, the non-dropped intermediate feature and gradient vectors are quantized using adaptive quantization levels determined based on the ranges of the vectors. To minimize the quantization error, the optimal quantization levels of this strategy are derived in a closed-form expression. Simulation results on the MNIST, CIFAR-100, and CelebA datasets demonstrate that SplitFC outperforms state-of-the-art SL frameworks by significantly reducing communication overheads while maintaining high accuracy.

Cited by

• Quantized-Tinyllava: a new multimodal foundation model enables efficient split learning
• Range Asymmetric Numeral Systems-Based Lightweight Intermediate Feature Compression for Split Computing of Deep Neural Networks
• Memory- and Latency-Constrained Inference of Large Language Models via Adaptive Split Computing
• Rate-Adaptive Semantic Communication via Multi-Stage Vector Quantization
• Communication and Computation Efficient Split Federated Learning in O-RAN

Related

• Mask-Encoded Sparsification: Mitigating Biased Gradients in Communication-Efficient Split Learning
• SL-ACC: A Communication-Efficient Split Learning Framework with Adaptive Channel-wise Compression
• Communication Efficient Split Learning of ViTs with Attention-based Double Compression
• C3-SL: Circular Convolution-Based Batch-Wise Compression for Communication-Efficient Split Learning
• Adaptive Compression for Communication-Efficient Distributed Training