Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks

2025/06/06 by Yuxuan Jiang, Jiang, Yuxuan, Ziming Zhou +8 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #Software Reliability and Analysis Research #Intelligent Tutoring Systems and Adaptive Learning

paper · pdf · doi:10.48550/arxiv.2506.14813

Abstract

Training deep learning (DL) models is a complex process, making it prone to silent errors that are challenging to detect and diagnose. This paper presents TRAINCHECK, a framework that takes a proactive checking approach to address silent training errors. TRAINCHECK automatically infers invariants tailored for DL training. It uses these invariants to proactively detect silent errors during the training process while providing debugging help. To evaluate TRAINCHECK, we reproduce 20 real-world silent training errors with diverse root causes. TRAINCHECK successfully detects 18 errors within a single training iteration. It also uncovers 6 unknown bugs in popular training libraries that lead to silent errors.

Cited by

• Role-Based Fault Tolerance System for LLM RL Post-Training
• Demystifying the Silence of Correctness Bugs in PyTorch Compiler
• Verifying Computational Graphs in Production-Grade Distributed Machine Learning Frameworks

Related

• TTrace: Lightweight Error Checking and Diagnosis for Distributed Training
• TFCheck : A TensorFlow Library for Detecting Training Issues in Neural\n Network Programs
• Training Set Debugging Using Trusted Items
• Data-Aware Training Quality Monitoring and Certification for Reliable Deep Learning
• DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs