在Google TPU-v3 Pod上缩放MLPerf-0.6模型
Scale MLPerf-0.6 models on Google TPU-v3 Pods
摘要：
最近Google TPU-v3 Pods提交给业界MLPerf v0.6训练基准测试，展示了一套行业相关ML模型的可扩展性。MLPerf定义了一套模型、数据集和基准测试时要遵循的规则，以确保结果在硬件、框架和公司之间具有可比性。使用这套模型，我们讨论了优化和技术，包括优化器的选择，空间分区和权重更新分片扩展到1024个TPU芯片。此外，我们确定的模型，使缩放它们具有挑战性的属性，如有限的数据并行和未缩放的权重。这些优化有助于在Google MLPerf-0.6提交中记录Transformer、Resnet-50和SSD的性能。
英文摘要：
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models. MLPerf defines a suite of models, datasets and rules to follow when benchmarking to ensure results are comparable across hardware, frameworks and companies. Using this suite of models, we discuss the optimizations and techniques including choice of optimizer, spatial partitioning and weight update sharding necessary to scale to 1024 TPU chips. Furthermore, we identify properties of models that make scaling them challenging, such as limited data parallelism and unscaled weights. These optimizations contribute to record performance in transformer, Resnet-50 and SSD in the Google MLPerf-0.6 submission.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
