NIXT: A NCCL Inspector Exporter Tool for Observability of Collective Communication in Large Model Training

2026/08/02 by Ziyang Jia, Sirshak Das, Jason Sewall +3
Computer Science · #cs.DC

paper · pdf

Accepted at IEEE IISWC 2026

arxiv created 2026/08/02 · arxiv updated 2026/08/04

Abstract

As machine learning workloads scale, it is increasingly important to gain more observability into the performance of collective communication to easily identify performance vari- ations and accelerate root cause identification. Towards this goal, the Nvidia Collective Communication Library (NCCL) introduced NCCL Inspector, a profiler plugin that provides lightweight and continuous reporting of NCCL communication performance statistics. However, the large volume of data collected by NCCL Inspector can be difficult to assess and to extract actionable insights from. This paper presents NIXT, a NCCL Inspector Exporter Tool that improves the observability of collective communication by providing readily accessible analysis and actionable insights from NCCL Inspector profiling. To highlight the benefits of our Exporter Tool, we present a case study of Nemotron-4 LLM pretraining on an Nvidia H100 GPU cluster with up to 2,048 GPUs, demonstrate observability into how communication phases change with ML parallelism and GPU scale, and perform attribution of performance variation and root cause analysis of stragglers.

Citations

• Robust LLM Training Infrastructure at ByteDance
• Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training
• Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks
• eACGM: Non-instrumented Performance Tracing and Anomaly Detection towards Machine Learning Systems
• Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications
• Understanding Stragglers in Large Model Training Using What-if Analysis
• Flare: Anomaly Diagnostics for Divergent LLM Training in GPU Clusters of Thousand-Plus Scale
• Mint: Cost-Efficient Tracing with All Requests Collection via Commonality and Variability Analysis
• DeepContext: A Context-aware, Cross-platform, and Cross-framework Tool for Performance Profiling and Analysis of Deep Learning Workloads
• Minder: Faulty Machine Detection for Large-scale Distributed Model Training
• Revisiting Reliability in Large-Scale Machine Learning Research Clusters
• Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization
• Characterization of Large Language Model Development in the Datacenter
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
• SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validation
• T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives
• Ring Attention with Blockwise Transformers for Near-Infinite Context
• Automatic Root Cause Analysis via Large Language Models for Cloud Incidents
• Comprehensive Deadlock Prevention for GPU Collective Communication
• The Benefit of Hindsight: Tracing Edge-Cases in Distributed Systems
• TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Horovod: fast and easy distributed deep learning in TensorFlow
• One weird trick for parallelizing convolutional neural networks