OpenDiLoCo:面向全球分布式低通信训练的开源框架
OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
AI总结:
本研究提出开源框架OpenDiLoCo,基于Hivemind库实现可复现的DiLoCo低通信大模型训练,经跨多国实验验证计算利用率达90-95%,支持FP16全规约梯度且无性能损失,可扩展至原工作3倍规模适配十亿参数模型。
摘要:
OpenDiLoCo是针对大语言模型的分布式低通信(Distributed Low-Communication, DiLoCo)训练方法的开源实现与复现版本。我们提供了可复现的DiLoCo实验实现,将其集成在基于Hivemind库的可扩展去中心化训练框架中。我们通过跨两大洲、三个国家训练模型验证了该框架的有效性,期间保持了90-95%的计算利用率。此外,我们针对算法的计算效率、工作节点数量的可扩展性开展了消融研究,证明其梯度可使用FP16进行全规约(all-reduce)且无性能下降。我们还将OpenDiLoCo扩展至原工作3倍的规模,验证了其对十亿参数级模型的适用性。
英文摘要:
OpenDiLoCo is an open-source implementation and replication of the Distributed Low-Communication (DiLoCo) training method for large language models. We provide a reproducible implementation of the DiLoCo experiments, offering it within a scalable, decentralized training framework using the Hivemind library. We demonstrate its effectiveness by training a model across two continents and three countries, while maintaining 90-95% compute utilization. Additionally, we conduct ablations studies focusing on the algorithm's compute efficiency, scalability in the number of workers and show that its gradients can be all-reduced using FP16 without any performance degradation. Furthermore, we scale OpenDiLoCo to 3x the size of the original work, demonstrating its effectiveness for billion parameter models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。