RailX: A Flexible, Scalable, and Low-Cost Network Architecture for Hyper-Scale LLM Training Systems

2025/07/25 by Yinxiao Feng, Tiancheng Chen, Feng, Yinxiao +13
Computer Science · #Parallel Computing and Optimization Techniques #Interconnection Networks and Systems #Graph Theory and Algorithms

paper · pdf · doi:10.48550/arxiv.2507.18889

Abstract

Increasingly large AI workloads are calling for hyper-scale infrastructure; however, traditional interconnection network architecture is neither scalable nor cost-effective enough. Tree-based topologies such as the Rail-optimized network are extremely expensive, while direct topologies such as Torus have insufficient bisection bandwidth and flexibility. In this paper, we propose RailX, a reconfigurable network architecture based on intra-node direct connectivity and inter-node circuit switching. Nodes and optical switches are physically 2D-organized, achieving better scalability than existing centralized circuit switching networks. We propose a novel interconnection method based on Hamiltonian Decomposition theory to organize separate rail-based rings into all-to-all topology, simultaneously optimizing ring-collective and all-to-all communication. More than 100K chips with hyper bandwidth can be interconnected with a flat switching layer, and the diameter is only 2∼4 inter-node hops. The network cost per injection/All-Reduce bandwidth of RailX is less than 10% of the Fat-Tree, and the cost per bisection/All-to-All bandwidth is less than 50% of the Fat-Tree. Specifically, only ∼$1.3B is required to interconnect 200K chips with 1.8TB bandwidth. RailX can also be used in the ML-as-a-service (MLaaS) scenario, where single or multiple training workloads with various shapes, scales, and parallelism strategies can be flexibly mapped, and failures can be worked around.

Citations

• Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects
• Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
• LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• Ring Attention with Blockwise Transformers for Near-Infinite Context
• DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
• CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters
• Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters
• Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problem
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
• HexaMesh: Scaling to Hundreds of Chiplets with an Optimized Chiplet Arrangement
• HammingMesh: A Network Topology for Large-Scale Deep Learning
• Mission Apollo: Landing Optical Circuit Switching at Datacenter Scale
• MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services
• Reducing Activation Recomputation in Large Transformer Models
• Air Quality in the New Delhi Metropolis under COVID-19 Lockdown
• TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs
• DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
• TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
• Scaling Vision with Sparse Mixture of Experts
• Efficient Large-Scale Language Model Training on GPU Clusters Using\n Megatron-LM
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Highly Available Data Parallel ML training on Mesh Networks
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• GLU Variants Improve Transformer
• Scaling Laws for Neural Language Models
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Related

• Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters
• UB-Mesh: a Hierarchically Localized nD-FullMesh Datacenter Network Architecture
• Photonic Rails in ML Datacenters
• Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure
• InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers