AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism

2025/11/05 by Wenyuan Xu, Chujie Chen, Xu, Wendong +19
Computer Science · #Cloud Computing and Resource Management #Distributed #FOS: Computer and information sciences #Parallel #Parallel Computing and Optimization Techniques #Software System Performance and Reliability #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2511.11617

openalex publication_date 2025/11/05 · openalex created_date 2025/11/19 · openalex updated_date 2026/07/28

Abstract

Large Language Model (LLM) inference services demand exceptionally high availability and low latency, yet multi-GPU Tensor Parallelism (TP) makes them vulnerable to single-GPU failures. We present AnchorTP, a state-preserving elastic TP framework for fast recovery. It (i) enables Elastic Tensor Parallelism (ETP) with unequal-width partitioning over any number of GPUs and compatibility with Mixture-of-Experts (MoE), and (ii) preserves model parameters and KV caches in GPU memory via a daemon decoupled from the inference process. To minimize downtime, we propose a bandwidth-aware planner based on a Continuous Minimal Migration (CMM) algorithm that minimizes reload bytes under a byte-cost dominance assumption, and an execution scheduler that pipelines P2P transfers with reloads. These components jointly restore service quickly with minimal data movement and without changing service interfaces. In typical failure scenarios, AnchorTP reduces Time to First Success (TFS) by up to 11x and Time to Peak (TTP) by up to 59% versus restart-and-reload.

Citations

• Cloud Native System for LLM Inference Serving
• All is Not Lost: LLM Recovery without Checkpoints
• Qwen3 Technical Report
• Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Training
• ReaLM: Reliable and Efficient Large Language Model Inference with Statistical Algorithm-Based Fault Tolerance
• Adaptive Fault Tolerance Mechanisms of Large Language Models in Cloud Computing Environments
• Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs
• DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
• Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studies
• DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving
• Mixtral of Experts
• SGLang: Efficient Execution of Structured Language Model Programs
• TRANSOM: An Efficient Fault-Tolerant System for Training LLMs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Adaptive Mixtures of Local Experts

Related

• FailSafe: High-performance Resilient Serving
• TP-Aware Dequantization
• PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling
• Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads
• KVDirect: Distributed Disaggregated LLM Inference