APMSqueeze：一种通信有效的Adam预处理动量SGD算法
APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm
摘要：
Adam是保证训练BERT和ImageNet等许多重要任务的效率和准确性的重要优化算法。然而，Adam通常与信息（梯度）压缩技术不兼容。因此，通信通常成为并行化Adam的瓶颈。本文提出了一种通过压缩梯度的误差补偿方法来提高通信效率的{ \bf A}DAM { \bf p}修正{ \bf M}动量SGD算法--APMSqueeze。所提出的算法实现了类似的收敛效率亚当的时期，但显着减少了每个时期的运行时间。在端到端性能方面（包括全精度预处理步骤），APMSqueeze能够提供{有时根据网络带宽，最高可达$2-10\times$的速度提升。}并对算法的收敛性和效率进行了理论分析。
英文摘要：
Adam is the important optimization algorithm to guarantee efficiency and accuracy for training many important tasks such as BERT and ImageNet. However, Adam is generally not compatible with information (gradient) compression technology. Therefore, the communication usually becomes the bottleneck for parallelizing Adam. In this paper, we propose a communication efficient {\bf A}DAM {\bf p}reconditioned {\bf M}omentum SGD algorithm-- named APMSqueeze-- through an error compensated method compressing gradients. The proposed algorithm achieves a similar convergence efficiency to Adam in term of epochs, but significantly reduces the running time per epoch. In terms of end-to-end performance (including the full-precision pre-condition step), APMSqueeze is able to provide {sometimes by up to $2-10\times$ speed-up depending on network bandwidth.} We also conduct theoretical analysis on the convergence and efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
