用于低延迟 AllReduce 的短路环
SHORTCUT: In-Collective Topology Reconfiguration for Low-Latency AllReduce
AI总结：
本文重新审视小消息 AllReduce 中 Ring 与 Recursive Doubling 的优劣，提出面向动态光子互连的集合内电路交换启发式调度，以降低传播延迟、拥塞和重配置开销。
摘要：
高效的集合通信对许多分布式 ML 和 HPC 应用至关重要。在这一背景下，人们普遍认为，用于 AllReduce 集合通信操作的 Ring 算法仅在大消息时是最优的，而 Recursive Doubling 更适合小消息，因为与 Ring 的线性步数相比，它具有对数级的步数。本文挑战了这一长期存在的假设，并表明：一旦考虑现实的传播延迟和链路容量约束，在基于环的 GPU 到 GPU 拓扑中，Ring 算法即使对于短消息也仍可保持最优。我们发现，Ring 和 Recursive Doubling 的总传播延迟本质上相加为相同值，但后者由于更长的跳数而产生显著更高的拥塞，从而导致完成时间增加。这一出人意料的结果激发了我们对集合内自适应拓扑的论证，尤其是在新兴光子互连背景下，它能够在集合通信粒度上突破静态拓扑设计的限制。我们为电路交换设计了一种简单而快速的启发式方法，使 Recursive Doubling 能够利用动态可重构的光子路径，谨慎平衡重配置延迟、传播延迟和链路拥塞，以最小化总体完成时间。我们使用现实的重配置延迟进行的初步评估表明，即使与静态环形拓扑上的 Ring AllReduce 相比，我们的电路交换调度也能使 Recursive Doubling 获得更快的完成时间。最后，我们强调了实现实用的集合内光子交换所面临的关键挑战和未来研究方向。
英文摘要：
Distributed ML training relies on efficient AllReduce communication to aggregate data across nodes. In this setting, reconfigurable optical interconnects offer high-bandwidth, energy-efficient direct links between accelerators but often produce ring-based topologies that remain static during a collective. The Ring AllReduce algorithm naturally matches these topologies but its cumulative latency grows linearly with node count. Low-latency algorithms such as Recursive Doubling (RD) instead achieve logarithmic cumulative per-step latency, but their long-distance exchanges incur dilation and congestion costs on a static ring. In-collective topology reconfiguration can eliminate these penalties but each topology change adds reconfiguration delay. The key question to improve AllReduce completion time is therefore not only how to reconfigure RD efficiently, but when selectively reconfigured RD becomes faster than the topology-matched Ring algorithm. We present Shortcut: an effective strategy for topology reconfiguration that enables RD to shortcut costly multi-hop communication only when it pays off - beyond the performance of the Ring algorithm. Across small and medium messages on 32 nodes, Shortcut achieves $4.4\times$-$6.0\times$ speedups over Ring. At 128 nodes, it remains up to $7\times$ faster with a $10\,μs$ reconfiguration delay, showing that selective reconfiguration is especially effective in latency-sensitive, large-scale settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
