OVERLORD: Ultimate Scaling of DataLoader for Multi-Source Large Foundation Model Training

2025/04/14 by Zhao, Juntao, Lu, Qi, Jia, Wei +14 · 3 citations
#Artificial Intelligence (cs.AI) #Distributed #FOS: Computer and information sciences #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.2504.09844

Abstract

Modern frameworks for training large foundation models (LFMs) employ dataloaders in a data-parallel manner, with each loader processing a disjoint subset of training data. Under multisource preprocessing, two fundamental challenges exist. First, due to the quadratic computational complexity of the attention operator, the non-uniform sample distribution over data-parallel ranks leads to significant workload imbalance among dataloaders, degrading the training efficiency. Second, supporting diverse data sources requires per-dataset file access states that are redundantly replicated across parallel loaders, consuming excessive memory. This also hinders dynamic data mixing (e.g., curriculum learning) and causes redundant access/memory overhead in hybrid parallelism. We present Omniload, an industrial-grade distributed data loading architecture for LFMs, with four innovations: (1) Disaggregated data preprocessing via role-specific actors (Source Loaders/Data Constructors) to eliminate source and parallelism redundant data access and ensure multisource scalability. (2) Centralized and declarative data plane for elastic multisource orchestration, such as long-short context, multimodality, and curriculum learning. (3) Multi-level auto-partitioning and scaling mechanism for source loaders under heterogeneous preprocessing costs. (4) Shadow loaders with differential checkpointing for fault recovery without workflow interruption. Deployed on production clusters scaling to multi-thousand GPUs, Omniload achieves: (1) 4.5x end-to-end training throughput improvement, (2) 13.5x reduction in CPU memory usage.

Citations

• Robust LLM Training Infrastructure at ByteDance
• Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training
• ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs
• The Streaming Batch Model for Efficient and Fault-Tolerant Heterogeneous Execution
• Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models
• Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
• The Llama 3 Herd of Models
• Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
• Characterization of Large Language Model Development in the Datacenter
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
• cedar: Optimized and Unified Machine Learning Input Data Pipelines
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
• Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
• Ring Attention with Blockwise Transformers for Near-Infinite Context
• Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models
• Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
• Extending Context Window of Large Language Models via Positional Interpolation
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• An Empirical Evaluation of Columnar Storage Formats
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Reducing Activation Recomputation in Large Transformer Models
• Pathways: Asynchronous Distributed Dataflow for ML
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• PyTorchVideo: A Deep Learning Library for Video Understanding
• The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
• tf.data: A Machine Learning Data Processing Framework
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch Distributed: Experiences on Accelerating Data Parallel Training
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Horovod: fast and easy distributed deep learning in TensorFlow
• Ray: A Distributed Framework for Emerging AI Applications
• Proximal Policy Optimization Algorithms
• ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development

Cited by

• Robust LLM Training Infrastructure at ByteDance

Related

• Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training
• ZeRO-Offload: Democratizing Billion-Scale Model Training
• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
• Dive into Big Model Training
• An Overview of the Data-Loader Landscape: Comparative Performance Analysis