Enhancing Cluster Scheduling in HPC: A Continuous Transfer Learning for Real-Time Optimization

2025/09/22 by Sliwko, Leszek, Mizera-Pietraszko, Jolanta
#Artificial Intelligence (cs.AI) #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.2509.22701

Abstract

This study presents a machine learning-assisted approach to optimize task scheduling in cluster systems, focusing on node-affinity constraints. Traditional schedulers like Kubernetes struggle with real-time adaptability, whereas the proposed continuous transfer learning model evolves dynamically during operations, minimizing retraining needs. Evaluated on Google Cluster Data, the model achieves over 99% accuracy, reducing computational overhead and improving scheduling latency for constrained tasks. This scalable solution enables real-time optimization, advancing machine learning integration in cluster management and paving the way for future adaptive scheduling strategies.

Citations

• Network Contention-Aware Cluster Scheduling with Reinforcement Learning
• The Case for Task Sampling based Learning for Cluster Job Scheduling
• Learning Scheduling Algorithms for Data Processing Clusters
• AGOCS -- Accurate Google Cloud Simulator Framework
• Borg, Omega, and Kubernetes
• A Survey on Transfer Learning
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Learning Scheduling Algorithms for Data Processing Clusters
• Cluster Workload Allocation: A Predictive Approach Leveraging Machine Learning Efficiency
• DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters
• Rubick: Exploiting Job Reconfigurability for Deep Learning Cluster Scheduling
• Large-scale Machine Learning Cluster Scheduling via Multi-agent Graph Reinforcement Learning