arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.07245cs.LGmath.OC
收藏

平滑矩阵优于平滑常数:分布式优化的更好通信压缩技术

Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimization

King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);Toyota Technological Institute at Chicago (TTIC)(芝加哥丰田技术研究所)

Mher Safaryan, Filip Hanzely, Peter Richtárik

摘要:

大规模分布式优化已经成为训练具有大量参数和训练数据的监督机器学习模型的默认工具。本领域的最新进展提供了用于加速训练的若干机制,包括{ \em压缩通信}、{ \em方差减小}和{ \em加速}。然而,这些方法中没有一种能够利用超出标准平滑常数的局部损失的固有丰富的数据依赖平滑结构。在本文中,我们认为,在训练监督模型时,{ \em平滑矩阵} -无处不在的平滑常数的信息丰富的概括-可以而且应该被利用,以进一步显着的收益,在理论和实践中。为了进一步减轻分布式优化中固有的通信负担,我们提出了一种新的通信稀疏化策略,可以充分利用与局部损失相关的平滑矩阵。为了展示这个工具的力量,我们描述了我们的稀疏化技术可以适应三个分布式优化算法- DCGD,DIANA和ADANA-产生显着节省的通信复杂性。新方法总是优于基线,通常是戏剧性的。

英文摘要:

Large scale distributed optimization has become the default tool for the training of supervised machine learning models with a large number of parameters and training data. Recent advancements in the field provide several mechanisms for speeding up the training, including {\em compressed communication}, {\em variance reduction} and {\em acceleration}. However, none of these methods is capable of exploiting the inherently rich data-dependent smoothness structure of the local losses beyond standard smoothness constants. In this paper, we argue that when training supervised models, {\em smoothness matrices} -- information-rich generalizations of the ubiquitous smoothness constants -- can and should be exploited for further dramatic gains, both in theory and practice. In order to further alleviate the communication burden inherent in distributed optimization, we propose a novel communication sparsification strategy that can take full advantage of the smoothness matrices associated with local losses. To showcase the power of this tool, we describe how our sparsification technique can be adapted to three distributed optimization algorithms -- DCGD, DIANA and ADIANA -- yielding significant savings in terms of communication complexity. The new methods always outperform the baselines, often dramatically so.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑