SuperNeurons：深度神经网络分布式训练中基于FFT的梯度稀疏化
SuperNeurons: FFT-based Gradient Sparsification in the Distributed Training of Deep Neural Networks
摘要：
深度神经网络分布式训练的性能和效率高度依赖于所有参与节点之间的梯度平均性能，而这又受到节点之间通信的限制。有两种主要的策略来减少通信开销：一种是通过将通信与计算重叠来隐藏通信，另一种是减少消息大小。第一种解决方案适用于线性神经架构，但ResNet和Inception等最新网络为这种重叠提供了有限的机会。因此，研究人员更加关注最小化沟通。在本文中，我们提出了一种新的梯度压缩框架，来自真正的梯度分布的见解，并取得了压缩比，精度和计算开销之间的平衡。我们的框架有两个主要的新组件：在频域中的梯度稀疏化，和一个基于范围的浮点表示来压缩和进一步压缩梯度频率。这两个组件都是动态的，具有可调参数，可根据精度要求和系统平台实现不同的压缩比，并在GPU上实现非常高的吞吐量。我们证明，我们的技术保证收敛与递减的压缩比。实验结果表明，该压缩框架在不影响精度和收敛速度的前提下，有效地提高了32 GPU集群上最流行的神经网络的可扩展性。
英文摘要：
The performance and efficiency of distributed training of Deep Neural Networks highly depend on the performance of gradient averaging among all participating nodes, which is bounded by the communication between nodes. There are two major strategies to reduce communication overhead: one is to hide communication by overlapping it with computation, and the other is to reduce message sizes. The first solution works well for linear neural architectures, but latest networks such as ResNet and Inception offer limited opportunity for this overlapping. Therefore, researchers have paid more attention to minimizing communication. In this paper, we present a novel gradient compression framework derived from insights of real gradient distributions, and which strikes a balance between compression ratio, accuracy, and computational overhead. Our framework has two major novel components: sparsification of gradients in the frequency domain, and a range-based floating point representation to quantize and further compress gradients frequencies. Both components are dynamic, with tunable parameters that achieve different compression ratio based on the accuracy requirement and systems' platforms, and achieve very high throughput on GPUs. We prove that our techniques guarantee the convergence with a diminishing compression ratio. Our experiments show that the proposed compression framework effectively improves the scalability of most popular neural networks on a 32 GPU cluster to the baseline of no compression, without compromising the accuracy and convergence speed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
