Mesh网络上的高可用数据并行ML训练
Highly Available Data Parallel ML training on Mesh Networks
摘要：
数据并行ML模型可能需要几天或几周的时间在几个加速器上进行训练。培训的长时间依赖于可用于作业的资源群集，以在整个持续时间内保持运行。在网状网络上，这是具有挑战性的，因为故障将在网格中创建漏洞。数据包必须绕过故障芯片才能实现完全连接。在本文中，我们提出的技术，路由梯度求和所有减少交通周围的芯片上的2-D网格。我们通过MLPerf-v0.7 ResNet-50和BERT基准测试来评估我们的容错allreduce技术的性能。性能结果显示，对512和1024 TPU-v3芯片上的训练吞吐量的影响最小。
英文摘要：
Data parallel ML models can take several days or weeks to train on several accelerators. The long duration of training relies on the cluster of resources to be available for the job to keep running for the entire duration. On a mesh network this is challenging because failures will create holes in the mesh. Packets must be routed around the failed chips for full connectivity. In this paper, we present techniques to route gradient summation allreduce traffic around failed chips on 2-D meshes. We evaluate performance of our fault tolerant allreduce techniques via the MLPerf-v0.7 ResNet-50 and BERT benchmarks. Performance results show minimal impact to training throughput on 512 and 1024 TPU-v3 chips.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
