StrataCL: Fabric-Native Communication Library for Production Supernodes

2026/07/29 by Tiancheng Hu, Jin Qin, Yuzheng Wang +14
Computer Science · #cs.DC

paper · pdf

arxiv created 2026/07/29 · arxiv updated 2026/07/30

Abstract

Modern distributed AI workloads run across hundreds of accelerators, making communication a major bottleneck. Existing communication libraries remain largely buffer-centric because user and communication buffers are managed separately, causing redundant data copies or costly user-buffer registration. This paper presents StrataCL, a zero-redundancy and fabric-native communication library for production supernodes. StrataCL introduces registration-on-allocation to realize user-buffer direct communication, and designs communication operators with workload-balanced NPU-core partitioning and NPU-driven SDMA offloading to exploit supernode architecture features. On the Huawei CloudMatrix384, StrataCL improves collective bus bandwidth by up to 1.6x and improves MoE dispatch/combine bus bandwidth by up to 1.4x. Across three production workloads, StrataCL improves LLM inference throughput by 1.9x, reduces P99 TTFT by 2.2x, and reduces LLM and Recsys training iteration time by 1.4x and 1.3x, respectively.

Citations

• Serving Large Language Models on Huawei CloudMatrix384
• FlashMoE: Fast Distributed MoE in a Single Kernel
• MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
• TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference
• Understanding Stragglers in Large Model Training Using What-if Analysis
• MSCCL++: Rethinking GPU Communication Abstractions for Cutting-edge AI Applications
• TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
• Demystifying the Communication Characteristics for Distributed Transformer Models
• Better & Faster Large Language Models via Multi-token Prediction
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
• T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• SGLang: Efficient Execution of Structured Language Model Programs
• Splitwise: Efficient generative LLM inference using phase splitting
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• Tutel: Adaptive Mixture-of-Experts at Scale
• No Provisioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computing
• Mixture-of-Experts with Expert Choice Routing
• TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs
• TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
• Clio: A Hardware-Software Co-Designed Disaggregated Memory System
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Related

• xDeepServe: Model-as-a-Service on Huawei CloudMatrix384
• Serving Large Language Models on Huawei CloudMatrix384
• UB-Mesh: a Hierarchically Localized nD-FullMesh Datacenter Network Architecture
• Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure
• SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips