当扩展失效时：网络和fabric效应对分布式GPU训练性能的影响
When Scaling Fails: Network and Fabric Effects on Distributed GPU Training Performance
AI总结：
本文研究了分布式GPU训练中网络和fabric效应导致扩展失效的原因，通过实证分析揭示了网络拓扑、拥堵动态等对训练性能的影响，并提出诊断原则以提高扩展性和可预测性。
摘要：
在分布式GPU训练中，通常假设增加节点数会带来可预测的性能提升。但在实践中，许多大规模部署在达到理论极限之前就遇到了回报递减和不稳定行为。本文探讨了为什么在真实系统中扩展会失效，重点研究了网络和fabric效应在其中的作用，这些效应常被高层训练框架所忽视。我们对多个生产规模集群中的分布式GPU训练性能进行了实证研究。我们的结果表明，一旦负载超过少量节点，网络拓扑、拥堵动态、集体同步行为以及GPU局部性经常主导端到端的训练性能。相同的模型和软件栈在不同的fabric设计和运行时通信模式下，可能会表现出截然不同的扩展特性。我们识别出当训练从单节点转为多节点执行时出现的反复失败模式，包括同步放大、拓扑引发的争用以及局部性驱动的性能变化。这些效应通常对标准分析工具不可见，因此常被误认为是框架或模型层面的低效。基于这些发现，我们概述了系统构建者可以应用的实用诊断原则，以理解扩展限制，提高可预测性，并减少大规模分布式训练的成本。
英文摘要：
Scaling distributed GPU training is commonly assumed to yield predictable performance gains as additional nodes are added. In practice, many large-scale deployments encounter diminishing returns and unstable behavior well before theoretical limits are reached. This paper examines why scaling fails in real systems, with a focus on the role of network and fabric effects that are often overlooked by higher-level training frameworks. We present an empirical study of distributed GPU training performance across multiple production-scale clusters. Our results show that network topology, congestion dynamics, collective synchronization behavior, and GPU locality frequently dominate end-to-end training performance once workloads move beyond a small number of nodes. Identical models and software stacks can exhibit sharply different scaling characteristics depending on fabric design and runtime communication patterns. We identify recurring failure modes that emerge as training transitions from single-node to multi-node execution, including synchronization amplification, topology-induced contention, and locality-driven performance variance. These effects are often invisible to standard profiling tools and are therefore misdiagnosed as framework or model-level inefficiencies. Based on these findings, we outline practical diagnostic principles that system builders can apply to understand scaling limits, improve predictability, and reduce the cost of large-scale distributed training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
