Large scale distributed optimization has become the default tool for the training of supervised machine learning models with a large number of parameters and training data. Recent advancements in the field provide several mechanisms for speeding up the training, including {\em compressed communication}, {\em variance reduction} and {\em acceleration}. However, none of these methods is capable of exploiting the inherently rich data-dependent smoothness structure of the local losses beyond standard smoothness constants. In this paper, we argue that when training supervised models, {\em smoothness matrices} -- information-rich generalizations of the ubiquitous smoothness constants -- can and should be exploited for further dramatic gains, both in theory and practice. In order to further alleviate the communication burden inherent in distributed optimization, we propose a novel communication sparsification strategy that can take full advantage of the smoothness matrices associated with local losses. To showcase the power of this tool, we describe how our sparsification technique can be adapted to three distributed optimization algorithms -- DCGD, DIANA and ADIANA -- yielding significant savings in terms of communication complexity. The new methods always outperform the baselines, often dramatically so.
On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach
University of Basel(巴塞尔大学);University of Oslo(奥斯陆大学);MBZUAI(马克斯·普朗克智能系统研究所);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);ELLIS Institute Tübingen(图宾根ELLIS研究所);Tübingen AI Center(图宾根人工智能中心)
Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm
Nanjing University(南京大学);Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学);Hong Kong University of Science and Technology(香港科学大学)
King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Institute of Science and Technology Austria(奥地利科学技术研究院)
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
Keio University(庆应大学);Stanford University(斯坦福大学);Midwestern University(中西部大学);California Institute of Technology(加州理工学院);Mila;Université de Montréal(蒙特利尔大学)