缓慢和陈旧的竞争者可以赢得比赛
Slow and Stale Gradients Can Win the Race
摘要：
分布式随机梯度下降（SGD）在以同步方式运行时，由于等待最慢的工人（掉队者），因此会在运行时延迟。异步方法可以减少落伍者，但会导致梯度失效，从而对收敛误差产生不利影响。在这项工作中，我们提出了一个新的理论表征的异步方法提供的加速比通过分析训练模型中的错误和实际的训练运行时（wallclock时间）之间的权衡。在我们的工作中的主要新颖之处是，我们的运行时分析考虑随机离散延迟，这有助于我们设计和比较分布式SGD算法，在离散和陈旧之间取得平衡。我们还提供了一个新的误差收敛分析的异步SGD变量没有界或指数延迟的假设。最后，基于我们的理论表征的错误-运行时的权衡，我们提出了一种方法，逐渐变化的同步分布式SGD和CIFAR 10数据集上展示其性能。
英文摘要：
Distributed Stochastic Gradient Descent (SGD) when run in a synchronous manner, suffers from delays in runtime as it waits for the slowest workers (stragglers). Asynchronous methods can alleviate stragglers, but cause gradient staleness that can adversely affect the convergence error. In this work, we present a novel theoretical characterization of the speedup offered by asynchronous methods by analyzing the trade-off between the error in the trained model and the actual training runtime(wallclock time). The main novelty in our work is that our runtime analysis considers random straggling delays, which helps us design and compare distributed SGD algorithms that strike a balance between straggling and staleness. We also provide a new error convergence analysis of asynchronous SGD variants without bounded or exponential delay assumptions. Finally, based on our theoretical characterization of the error-runtime trade-off, we propose a method of gradually varying synchronicity in distributed SGD and demonstrate its performance on CIFAR10 dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
