PRISM: Probabilistic Runtime Insights and Scalable Performance Modeling for Large-Scale Distributed Training

2025/10/17 by Alicia Golden, Golden, Alicia, Michael Kuchnik +11
Computer Science · #Advanced Neural Network Applications #Distributed #FOS: Computer and information sciences #Graph Theory and Algorithms #Parallel #Parallel Computing and Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2510.15596

openalex publication_date 2025/10/17 · openalex created_date 2025/10/21 · openalex updated_date 2026/07/28

Abstract

Large model training beyond tens of thousands of GPUs is an uncharted territory. At such scales, disruptions to the training process are not a matter of if, but a matter of when -- a stochastic process degrading training productivity. Dynamic runtime variation will become increasingly more frequent as training scales up and as GPUs are operated in increasingly power-limited and thermally-stressed environments. At the 64,000+ GPU scale, we already observe 9% GPU time variability for frontier foundation model training. Motivated by our analysis and the large design space around performance variability, we present PRISM -- a performance modeling framework that captures the stochastic nature of large-scale distributed training. The core of PRISM is a statistical method that quantifies probabilistic guarantees on training time. Using PRISM, we explore the design and optimization space of distributed training, enabling principled, variability-aware decisions that improve performance and system efficiency at scale.

Citations

• Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective
• "Energon": Unveiling Transformers from GPU Power and Thermal Side-Channels
• Cooling Matters: Benchmarking Large Language Models and Vision-Language Models on Liquid-Cooled Versus Air-Cooled H100 GPU Systems
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training
• Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
• Understanding Stragglers in Large Model Training Using What-if Analysis
• Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training
• Context Parallelism for Scalable Million-Token Inference
• Revisiting Reliability in Large-Scale Machine Learning Research Clusters
• FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Training
• Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects
• Is Flash Attention Stable?
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
• SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validation
• Credence: Augmenting Datacenter Switch Buffer Sharing with ML Predictions
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Zero Bubble Pipeline Parallelism
• vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training
• MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems
• Comparative Analysis of CPU and GPU Profiling for Deep Learning Models
• CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters
• Proteus: Simulating the Performance of Distributed DNN Training
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation
• Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich Systems
• Efficient Large-Scale Language Model Training on GPU Clusters Using\n Megatron-LM
• A Runtime-Based Computational Performance Predictor for Deep Neural Network Training
• TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models
• Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
• Beyond Data and Model Parallelism for Deep Neural Networks

Related

• Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective
• Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training Inefficiency
• Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms
• Large-Scale Stochastic Learning using GPUs
• Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers