A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Network

2026/03/31 by Aojie Jiang, Kang Zhu, Zhiheng Zhang +4
Computer Science · #cs.AR

paper · pdf · doi:10.48550/arxiv.2603.28239

arxiv created 2026/08/04 · arxiv updated 2026/08/05

Abstract

Tensor parallelism (TP) has become a key technique for latency-sensitive LLM inference, but it introduces frequent, tightly synchronized All-Reduce operations that lie directly on the inference critical path, making inter-GPU communication a major performance bottleneck. In-network computing offers a promising approach to mitigating this bottleneck. However, we observe that the existing solution, NVLink SHARP (NVLS), is fundamentally constrained by its GPU-driven execution model that exposes in-network computing as GPU-issued, element-granular memory operations rather than a switch-orchestrated collective. This design forces redundant switch-GPU-switch data movement and cannot perform scale-aware quantized All-Reduce due to its lack of cross-element coordination. To address these limitations, we propose SCIN, the first switch-centric in-network computing architecture for multi-accelerator shared-memory networks. SCIN replaces GPU-driven collective orchestration with an in-switch accelerator (ISA) that directly initiates memory transactions and orchestrates the complete All-Reduce operation. This switch-driven execution eliminates redundant data movement inherent in NVLS, shortens synchronization paths, and fully offloads collective execution from GPUs. Furthermore, centralized ISA control enables cross-element coordination between quantized data and their scale factors. SCIN exploits this capability through a pipelined dequantization-reduction-requantization datapath, enabling scale-aware in-network quantized All-Reduce and nearly halving communication volume with negligible accuracy degradation. We implement SCIN on a multi-FPGA prototype to validate its feasibility. For a simulated 8-GPU system, SCIN accelerates small- and large-message All-Reduce by up to 1.8x and 2.6x over NVLS, respectively, translating into up to 1.12x TPOT and 1.42x TTFT speedups.

Citations

• Challenges and Research Directions for Large Language Model Inference Hardware
• Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guarantees
• EQuARX: Efficient Quantized AllReduce in XLA for Distributed Machine Learning Acceleration
• TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference
• Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference
• Communication Compression for Tensor Parallel LLM Inference
• Context Parallelism for Scalable Million-Token Inference
• Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant
• NanoFlow: Towards Optimal Large Language Model Serving Throughput
• Mirage: A Multi-Level Superoptimizer for Tensor Programs
• Vidur: A Large-Scale Simulation Framework For LLM Inference
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• SGLang: Efficient Execution of Structured Language Model Programs
• LLM-FP4: 4-Bit Floating-Point Quantized Transformers
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
• Measuring Massive Multitask Language Understanding
• PIQA: Reasoning about Physical Commonsense in Natural Language
• MLPerf Inference Benchmark
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Scaling Distributed Machine Learning with In-Network Aggregation
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Attention Is All You Need
• Pointer Sentinel Mixture Models