Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models

2024/11/01 by Runsheng Benson Guo, Utkarsh Anand, Guo, Runsheng Benson +5 · 3 citations
Computer Science · #Distributed #FOS: Computer and information sciences #Parallel #Topic Modeling #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2411.01075

openalex publication_date 2024/11/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Training transformer models requires substantial GPU compute and memory resources. In homogeneous clusters, distributed strategies allocate resources evenly, but this approach is inefficient for heterogeneous clusters, where GPUs differ in power and memory. As high-end GPUs are costly and limited in availability, heterogeneous clusters with diverse GPU types are becoming more common. Existing methods attempt to balance compute across GPUs based on capacity but often underutilize compute due to memory constraints. We present Cephalo, a system that optimizes compute and memory usage by decoupling compute distribution from training state assignment. Cephalo outperforms state-of-the-art methods by achieving significantly higher training throughput while supporting larger models and batch sizes.

Cited by

• Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters
• Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations
• Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters

Related

• Memory and Bandwidth are All You Need for Fully Sharded Data Parallel
• Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading
• Galvatron: Automatic Distributed Training for Large Transformer Models
• HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models
• Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters