ATOMO：通过原子稀疏化实现通信高效学习
ATOMO: Communication-efficient Learning via Atomic Sparsification
摘要：
由于计算节点之间频繁传输梯度更新，分布式模型训练遭受通信开销。为了减轻这些开销，一些研究提出了使用稀疏随机梯度。我们认为，这些方面的一般稀疏化方法，可以操作任何可能的原子分解。值得注意的例子包括元素方式，奇异值和傅立叶分解。我们提出了原子稀疏随机梯度，一个一般的框架。给定梯度、原子分解和稀疏预算，ATOMO给出原子的随机无偏稀疏化，最小化方差。我们表明，最近的方法，如QSGD和TernGrad是ATOMO的特例，稀疏化神经网络梯度的奇异值分解，而不是它们的坐标，可以导致更快的分布式训练。
英文摘要：
Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any possible atomic decomposition. Notable examples include element-wise, singular value, and Fourier decompositions. We present ATOMO, a general framework for atomic sparsification of stochastic gradients. Given a gradient, an atomic decomposition, and a sparsity budget, ATOMO gives a random unbiased sparsification of the atoms minimizing variance. We show that recent methods such as QSGD and TernGrad are special cases of ATOMO and that sparsifiying the singular value decomposition of neural networks gradients, rather than their coordinates, can lead to significantly faster distributed training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
