Quantize-Sample-and-Verify: LLM Acceleration via Adaptive Edge-Cloud Speculative Decoding

2025/07/01 by Guangyi Zhang, Yunlong Cai, Zhang, Guangyi +7
Computer Science · Engineering · #IoT and Edge/Fog Computing #Advanced Neural Network Applications #Advanced MIMO Systems Optimization

paper · pdf · doi:10.48550/arxiv.2507.00605

Abstract

In edge-cloud speculative decoding (SD), edge devices equipped with small language models (SLMs) generate draft tokens that are verified by large language models (LLMs) in the cloud. A key bottleneck in such systems is the limited communication bandwidth between edge and cloud, which necessitates quantization of the information transmitted about generated tokens. In this work, we introduce a novel quantize-sample (Q-S) strategy that provably preserves the output distribution of the cloud-based model, ensuring that the verified tokens match the distribution of those that would have been generated directly by the LLM. We develop a throughput model for edge-cloud SD that explicitly accounts for communication latency. Leveraging this model, we propose an adaptive mechanism that optimizes token throughput by dynamically adjusting the draft length and quantization precision in response to both semantic uncertainty and channel conditions. Simulations demonstrate that the proposed Q-S approach significantly improves decoding efficiency in realistic edge-cloud deployment scenarios.

Citations

• Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits
• Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models
• AI Flow at the Network Edge
• Vector Quantization Prompting for Continual Learning
• Quantized Prompt for Efficient Generalization of Vision-Language Models
• ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
• Latency-Distortion Tradeoffs in Communicating Classification Results over Noisy Channels
• Accelerating Large Language Model Decoding with Speculative Sampling
• Fast Inference from Transformers via Speculative Decoding
• OPT: Open Pre-trained Transformer Language Models
• Multi-RAT for IoT: The Potential in Combining LoRaWAN and NB-IoT
• Deep Reinforcement Learning with Double Q-learning

Related

• Conformal Sparsification for Bandwidth-Efficient Edge-Cloud Speculative Decoding
• SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
• DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving
• DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decoding
• GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inference