arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.18512cs.CL
收藏

具有重叠通信的流式 DiLoCo:迈向分布式免费午餐

Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch

Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院);Apple(苹果公司);Google(谷歌公司)
Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, Alexandre … 展开作者

Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc'Aurelio Ranzato, Paul Barham

AI总结:

本文改进了DiLoCo分布式算法,通过按序列同步参数子集、允许同步期间继续训练以及量化交换数据,在保持模型质量的同时将跨workers的所需带宽降低了两个数量级。

摘要:

大型语言模型(LLMs)的训练通常分布在大量加速器上以减少训练时间。由于在每个梯度步骤都需要交换内部状态和参数梯度,所有设备需要使用低延迟高带宽的通信链路进行同址部署,以支持所需的大量比特交换。最近,像 DiLoCo 这样的分布式算法放宽了这种同址约束:加速器可以被分组为“workers”,其中 workers 之间的同步仅不频繁地发生。这进而意味着 workers 可以承受由较低带宽的通信链路连接,而不影响学习质量。然而,在这些方法中,跨 workers 的通信仍然需要与以前相同的峰值带宽,因为同步要求所有参数在所有 workers 之间交换。在本文中,我们从三个方面改进了 DiLoCo。首先,我们按顺序仅同步参数子集,而不是一次性全部同步,这大大降低了峰值带宽。其次,我们允许 workers 在同步期间继续训练,这减少了实际运行时间。第三,我们对 workers 交换的数据进行量化,这进一步降低了跨 workers 的带宽。通过适当地结合这些修改,我们在实验中表明,我们可以分布式训练十亿规模的参数并达到与以前相似的质量,但所需带宽减少了两个数量级。

英文摘要:

Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time. Since internal states and parameter gradients need to be exchanged at each and every single gradient step, all devices need to be co-located using low-latency high-bandwidth communication links to support the required high volume of exchanged bits. Recently, distributed algorithms like DiLoCo have relaxed such co-location constraint: accelerators can be grouped into ``workers'', where synchronizations between workers only occur infrequently. This in turn means that workers can afford being connected by lower bandwidth communication links without affecting learning quality. However, in these methods, communication across workers still requires the same peak bandwidth as before, as the synchronizations require all parameters to be exchanged across all workers. In this paper, we improve DiLoCo in three ways. First, we synchronize only subsets of parameters in sequence, rather than all at once, which greatly reduces peak bandwidth. Second, we allow workers to continue training while synchronizing, which decreases wall clock time. Third, we quantize the data exchanged by workers, which further reduces bandwidth across workers. By properly combining these modifications, we show experimentally that we can distribute training of billion-scale parameters and reach similar quality as before, but reducing required bandwidth by two orders of magnitude.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑