Ethereal：大规模分布式训练中的分而治之网络负载均衡
Ethereal: Divide and Conquer Network Load Balancing in Large-Scale Distributed Training
AI总结：
本文挑战数据包喷洒必要性，提出单路径传输配合流拆分可接近其性能，并设计Ethereal算法，在链路故障下较数据包喷洒和REPS分别降低完成时间30%和40%。
摘要：
生产数据中心中的大规模分布式训练构成了一个具有挑战性的工作负载，其瓶颈在于网络通信。为此，主要的行业参与者（例如超以太网联盟）和部分学术界人士出人意料地几乎一致认为，数据包喷洒对于提升大规模分布式训练工作负载的性能是必要的。在本文中，我们挑战了这一主流观点，并提出问题：单路径传输在性能上能多接近数据包喷洒？我们证明，从网卡的角度来看，单路径传输是足够的，并且其性能几乎可以与理想的数据包喷洒相媲美，尤其是在基于CLOS拓扑的分布式训练场景中。我们的论断基于对由集合通信模式驱动的工作负载的四个关键观察：（i）流在到达时大小已知；（ii）在集合通信的每一步内，流的大小相等；（iii）集合通信的完成时间比单个流的完成时间更为关键；（iv）流可以在到达时被拆分，从而直接从应用层控制负载均衡。我们提出了Ethereal，一种简单的分布式负载均衡算法，它机会性地拆分流，并以透明的方式为每个流分配路径，对现有RDMA网卡几乎无需修改。我们使用Astra-Sim进行的评估涵盖了广泛的集合通信算法和GPT模型，结果表明，即使在链路故障的情况下，与数据包喷洒相比，Ethereal显著减少了完成时间，最多减少30%，与REPS相比最多减少40%。本文为开发面向大规模分布式训练的下一代传输协议提供了另一种视角。
英文摘要：
Large-scale distributed training in production datacenters constitutes a challenging workload bottlenecked by network communication. In response, both major industry players (e.g., Ultra Ethernet Consortium) and parts of academia have surprisingly, and almost unanimously, agreed that packet spraying is \emph{necessary} to improve the performance of large-scale distributed training workloads. In this paper, we challenge this prevailing belief and pose the question: \emph{How close can singlepath transport come to matching the performance of packet spraying?} We demonstrate that singlepath transport (from a NIC's perspective) is sufficient and can perform nearly as well as ideal packet spraying, particularly in the context of distributed training in CLOS-based topologies. Our assertion is based on four key observations about workloads driven by collective communication patterns: \emph{(i)} flow sizes are known upon arrival, \emph{(ii)} flow sizes are equal within each step of a collective, \emph{(iii)} the completion time of a collective is more critical than individual flow completion times, and \emph{(iv)} flows can be \emph{split} upon arrival to control load balancing directly from the application layer. We present Ethereal, a simple distributed load balancing algorithm that opportunistically splits flows and assigns paths to each flow in a transparent manner, requiring little to no changes to existing RDMA NICs. Our evaluation, spanning a wide range of collective communication algorithms and GPT models using Astra-Sim, shows that Ethereal significantly reduces the completion times by up to $30\%$ compared to packet spraying and by up to $40\%$ compared to REPS, even under link failures. This paper offers an alternative perspective for developing next-generation transport protocols tailored to large-scale distributed training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
