ShadowSync：在后台执行同步以实现高度可扩展的分布式训练
ShadowSync: Performing Synchronization in the Background for Highly Scalable Distributed Training
摘要：
推荐系统通常使用大量数据进行训练，而分布式训练是缩短训练时间的主力。虽然训练吞吐量可以通过简单地添加更多的工人来增加，但保持模型质量也越来越具有挑战性。在本文中，我们提出了\shadowsync，一个分布式框架，专门为现代规模的推荐系统培训。与以前的同步作为训练过程的一部分发生的工作相比，\shadowsync将同步与训练分离，并在后台运行。这种隔离显著降低了同步开销并增加了同步频率，因此我们在大规模训练时能够获得高吞吐量和出色的模型质量。通过训练深度神经网络进行点击率预测任务的实验，证实了我们的过程的优越性。我们的框架是能够表达数据并行性和/或模型并行，通用主机的各种类型的同步算法，并很容易适用于大规模的问题在其他领域。
英文摘要：
Recommendation systems are often trained with a tremendous amount of data, and distributed training is the workhorse to shorten the training time. While the training throughput can be increased by simply adding more workers, it is also increasingly challenging to preserve the model quality. In this paper, we present \shadowsync, a distributed framework specifically tailored to modern scale recommendation system training. In contrast to previous works where synchronization happens as part of the training process, \shadowsync separates the synchronization from training and runs it in the background. Such isolation significantly reduces the synchronization overhead and increases the synchronization frequency, so that we are able to obtain both high throughput and excellent model quality when training at scale. The superiority of our procedure is confirmed by experiments on training deep neural networks for click-through-rate prediction tasks. Our framework is capable to express data parallelism and/or model parallelism, generic to host various types of synchronization algorithms, and readily applicable to large scale problems in other areas.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
