99%的分布式优化是浪费时间：问题以及如何解决它
99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it
摘要：
许多用于训练机器学习模型的流行的分布式优化方法都符合以下模板：每个工作者独立计算局部梯度估计，然后将其传送给主机，主机随后执行平均。平均值被广播回工作进程，工作进程使用它来执行梯度类型的步骤，以更新模型的本地版本。众所周知，许多这样的方法，包括SGD、SAGA和用于过参数化模型的加速SGD，不能很好地随并行工作器的数量而扩展。在本文中，我们观察到上述模板从根本上是低效的，因为工作人员不必要地传达了太多的数据，这会减慢整个系统的速度。我们提出了一个基于我们在这项工作中开发的新的更新稀疏化方法的修复，我们建议将其用于现有方法之上。也就是说，我们开发了一个新的变种的并行块坐标下降的基础上独立稀疏的局部梯度估计通信之前。我们证明，只有$m/n$块发送的每个$n$工人，其中$m$是参数块的总数，底层分布式方法的理论迭代复杂性基本上不受影响。作为说明，这意味着当使用$n=100$并行工作器时，$99\%$块的通信是冗余的，因此是浪费时间。我们的理论主张得到了广泛的数值实验的支持，这些实验在许多合成和真实数据集上证明了与我们的理论几乎完美的匹配。
英文摘要：
Many popular distributed optimization methods for training machine learning models fit the following template: a local gradient estimate is computed independently by each worker, then communicated to a master, which subsequently performs averaging. The average is broadcast back to the workers, which use it to perform a gradient-type step to update the local version of the model. It is also well known that many such methods, including SGD, SAGA, and accelerated SGD for over-parameterized models, do not scale well with the number of parallel workers. In this paper we observe that the above template is fundamentally inefficient in that too much data is unnecessarily communicated by the workers, which slows down the overall system. We propose a fix based on a new update-sparsification method we develop in this work, which we suggest be used on top of existing methods. Namely, we develop a new variant of parallel block coordinate descent based on independent sparsification of the local gradient estimates before communication. We demonstrate that with only $m/n$ blocks sent by each of $n$ workers, where $m$ is the total number of parameter blocks, the theoretical iteration complexity of the underlying distributed methods is essentially unaffected. As an illustration, this means that when $n=100$ parallel workers are used, the communication of $99\%$ blocks is redundant, and hence a waste of time. Our theoretical claims are supported through extensive numerical experiments which demonstrate an almost perfect match with our theory on a number of synthetic and real datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
