SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs

2025/05/16 by Park, Jinwoo, Cho, Seunggeun, Han, Dongsu · 2 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.17052

Abstract

Large language models (LLMs) power many modern applications, but serving them at scale remains costly and resource-intensive. Current server-centric systems overlook consumer-grade GPUs at the edge. We introduce SpecEdge, an edge-assisted inference framework that splits LLM workloads between edge and server GPUs using a speculative decoding scheme, exchanging only token outputs over the network. SpecEdge employs proactive edge drafting to overlap edge token creation with server verification and pipeline-aware scheduling that interleaves multiple user requests to increase server-side throughput. Experiments show SpecEdge enhances overall cost efficiency by 1.91x through achieving 2.22x server throughput, and reduces inter token latency by 11.24% compared to a server-only baseline, introducing a scalable, cost-effective paradigm for LLM serving. The code is available at https://github.com/kaist-ina/specedge

Citations

• AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration
• PEARL: Parallel Speculative Decoding with Adaptive Draft Length
• The Llama 3 Herd of Models
• PowerInfer-2: Fast Large Language Model Inference on a Smartphone
• SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
• StarCoder 2 and The Stack v2: The Next Generation
• Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
• Speculative Streaming: Fast LLM Inference without Auxiliary Models
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
• TinyLlama: An Open-Source Small Language Model
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
• Distributed Inference and Fine-tuning of Large Language Models Over The Internet
• SpecTr: Fast Speculative Decoding via Optimal Transport
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
• Mistral 7B
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• OpenAssistant Conversations -- Democratizing Large Language Model Alignment
• GPT-4 Technical Report
• Accelerating Large Language Model Decoding with Speculative Sampling
• Fast Inference from Transformers via Speculative Decoding
• Efficiently Scaling Transformer Inference
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• ZeRO-Offload: Democratizing Billion-Scale Model Training
• Language Models are Few-Shot Learners
• Training Large Neural Networks with Constant Memory using a New Execution Algorithm
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing
• Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
• Attention Is All You Need
• Pointer Sentinel Mixture Models
• Deep Residual Learning for Image Recognition
• You Only Look Once: Unified, Real-Time Object Detection

Cited by

• Fast LLM Post-training via Decoupled and Fastest-of-N Speculation
• CoEdge-RAG: Optimizing Hierarchical Scheduling for Retrieval-Augmented LLMs in Collaborative Edge Computing

Related

• SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
• EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
• iServe: An Intent-based Serving System for LLMs
• CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
• DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decoding