AI 中文总结
本文提出NIXT工具，解决NCCL Inspector数据量大难以分析的问题，通过案例展示其在大模型训练中提升集合通信可观测性、定位性能问题的作用。
AI 中文摘要
随着机器学习工作负载规模不断扩大，对集合通信性能提升可观测性变得愈发重要，以便轻松识别性能波动并加速根本原因定位。为实现这一目标，英伟达集合通信库（NCCL）推出了NCCL Inspector，这是一款分析器插件，可提供NCCL通信性能统计数据的轻量级、持续报告。然而，NCCL Inspector收集的大量数据难以评估和提取可操作的见解。本文提出了NIXT，一款NCCL Inspector导出工具，通过从NCCL Inspector分析中提供易于访问的分析和可操作见解，提升集合通信的可观测性。为突出该导出工具的优势，本文展示了在英伟达H100 GPU集群（最多2048个GPU）上进行Nemotron-4大语言模型预训练的案例研究，证明了对通信阶段如何随机器学习并行性和GPU规模变化的可观测性，并完成了性能波动的归因和掉队者的根本原因分析。
英文摘要
As machine learning workloads scale, it is increasingly important to gain more observability into the performance of collective communication to easily identify performance vari- ations and accelerate root cause identification. Towards this goal, the Nvidia Collective Communication Library (NCCL) introduced NCCL Inspector, a profiler plugin that provides lightweight and continuous reporting of NCCL communication performance statistics. However, the large volume of data collected by NCCL Inspector can be difficult to assess and to extract actionable insights from. This paper presents NIXT, a NCCL Inspector Exporter Tool that improves the observability of collective communication by providing readily accessible analysis and actionable insights from NCCL Inspector profiling. To highlight the benefits of our Exporter Tool, we present a case study of Nemotron-4 LLM pretraining on an Nvidia H100 GPU cluster with up to 2,048 GPUs, demonstrate observability into how communication phases change with ML parallelism and GPU scale, and perform attribution of performance variation and root cause analysis of stragglers.
CommentsAccepted at IEEE IISWC 2026
