Lagom：释放通信与计算重叠在分布式大模型训练中的潜力
Lagom: Unleashing the Power of Communication and Computation Overlapping for Distributed LLM Training
AI总结：
Lagom通过统一成本模型和优先级搜索算法，优化分布式大模型训练中的通信与计算重叠，实现速度提升。
摘要：
在分布式大模型训练中，通信与计算的重叠至关重要，但优化这一过程，尤其是在计算成为瓶颈时，仍然具有挑战性。我们提出了Lagom系统，该系统共同调节通信参数以平衡计算与通信之间的资源使用。通过引入统一的成本模型和基于优先级的搜索算法，Lagom将优化复杂度从指数级降低到线性级。在高带宽和低带宽GPU集群上的评估表明，Lagom在不同模型和并行化方案上分别实现了1.07-1.33倍和1.03-1.27倍的速度提升，优于NCCL和AutoCCL。
英文摘要：
Overlapping communication with computation is crucial for distributed large-model training, yet optimizing it - especially when computation becomes the bottleneck-remains challenging. We present Lagom, a system that co-tunes communication parameters to balance resource usage between computation and communication. By introducing a unified cost model and a priority-based search algorithm, Lagom reduces optimization complexity from exponential to linear. Evaluations on high- and low-bandwidth GPU clusters show that Lagom achieves 1.07-1.33x and 1.03-1.27x speedup over NCCL and AutoCCL across diverse models and parallelizations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
