分布式随机优化算法能够同时处理大规模数据集，显著加快模型训练。然而，其有效性往往受到分布式网络稀疏性和数据异构性的阻碍。本文提出一种动量加速的分布式随机梯度算法，称为 Exact-Diffusion with Momentum（EDM），该算法可缓解数据异构性带来的偏差，并结合深度学习中常用的动量技术以提高收敛速度。理论分析表明，当应用于非凸目标函数时，EDM 算法以次线性速率收敛到最优解的邻域，且该邻域半径与数据异构性无关；在比强凸性更弱的 Polyak-Łojasiewicz 条件下，它以线性速率收敛到目标区域。本文用于处理复杂分布式参数更新结构中动量项的分析技术给出了足够紧的收敛上界，为其他基于动量的分布式算法的理论分析提供了新视角。
英文摘要：
Distributed stochastic optimization algorithms can simultaneously process large-scale datasets, significantly accelerating model training. However, their effectiveness is often hindered by the sparsity of distributed networks and data heterogeneity. In this paper, we propose a momentum-accelerated distributed stochastic gradient algorithm, termed Exact-Diffusion with Momentum (EDM), which mitigates the bias from data heterogeneity and incorporates momentum techniques commonly used in deep learning to enhance convergence rate. Our theoretical analysis demonstrates that the EDM algorithm converges sub-linearly to the neighborhood of the optimal solution, the radius of which is irrespective of data heterogeneity, when applied to non-convex objective functions; under the Polyak-Lojasiewicz condition, which is a weaker assumption than strong convexity, it converges linearly to the target region. Our analysis techniques employed to handle momentum in complex distributed parameter update structures yield a sufficiently tight convergence upper bound, offering a new perspective for the theoretical analysis of other momentum-based distributed algorithms.
Alibaba Group（阿里巴巴集团）；University of Pennsylvania（宾夕法尼亚大学）
The scale of deep learning nowadays calls for efficient distributed training algorithms. Decentralized momentum SGD (DmSGD), in which each node averages only with its neighbors, is more communication efficient than vanilla Parallel momentum SGD that incurs global average across all computing nodes. On the other hand, the large-batch training has been demonstrated critical to achieve runtime speedup. This motivates us to investigate how DmSGD performs in the large-batch scenario. In this work, we find the momentum term can amplify the inconsistency bias in DmSGD. Such bias becomes more evident …
Distributed descent-based methods are an essential toolset to solving optimization problems in multi-agent system scenarios. Here the agents seek to optimize a global objective function through mutual cooperation. Oftentimes, cooperation is achieved over a wireless communication network that is prone to delays and errors. There are many scenarios wherein the objective function is either non-differentiable or merely observable. In this paper, we present a cross-entropy based distributed stochastic approximation algorithm (SA) that finds a minimum of the objective, using only samples. We call th…
Xi’an Jiaotong University（西安交通大学）；Fudan University（复旦大学）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
Emerging distributed applications recently boosted the development of decentralized machine learning, especially in IoT and edge computing fields. In real-world scenarios, the common problems of non-convexity and data heterogeneity result in inefficiency, performance degradation, and development stagnation. The bulk of studies concentrates on one of the issues mentioned above without having a more general framework that has been proven optimal. To this end, we propose a unified paradigm called UMP, which comprises two algorithms, D-SUM and GT-DSUM, based on the momentum technique with decentra…
Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization
Department of Mathematical Sciences（数学科学系）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；Department of Computer Science（计算机科学系）；MIT-IBM Watson AI Lab, IBM Research（MIT-IBM Watson AI实验室，IBM研究）；IBM T. J. Watson Research Center（IBM T.J. 华生研究中心）
Georgia Institute of Technology（佐治亚理工学院）；Virginia Tech（弗吉尼亚理工大学）
We study a decentralized variant of stochastic approximation, a data-driven approach for finding the root of an operator under noisy measurements. A network of agents, each with its own operator and data observations, cooperatively find the fixed point of the aggregate operator over a decentralized communication graph. Our main contribution is to provide a finite-time analysis of this decentralized stochastic approximation method when the data observed at each agent are sampled from a Markov process; this lack of independence makes the iterates biased and (potentially) unbounded. Under fairly …
We consider decentralized stochastic optimization with the objective function (e.g. data samples for machine learning task) being distributed over $n$ machines that can only communicate to their neighbors on a fixed communication graph. To reduce the communication bottleneck, the nodes compress (e.g. quantize or sparsify) their model updates. We cover both unbiased and biased compression operators with quality denoted by $ω\leq 1$ ($ω=1$ meaning no compression). We (i) propose a novel gossip-based stochastic gradient descent algorithm, CHOCO-SGD, that converges at rate $\mathcal{O}\left(1/(nT)…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；University of British Columbia（不列颠哥伦比亚大学）
Decentralized learning offers privacy and communication efficiency when data are naturally distributed among agents communicating over an underlying graph. Motivated by overparameterized learning settings, in which models are trained to zero training loss, we study algorithmic and generalization properties of decentralized learning with gradient descent on separable data. Specifically, for decentralized gradient descent (DGD) and a variety of loss functions that asymptote to zero at infinity (including exponential and logistic losses), we derive novel finite-time generalization bounds. This co…
The University of Queensland（昆士兰大学）；Griffith University（格里菲斯大学）；Monash University（莫纳什大学）
Actuated by the growing attention to personal healthcare and the pandemic, the popularity of E-health is proliferating. Nowadays, enhancement on medical diagnosis via machine learning models has been highly effective in many aspects of e-health analytics. Nevertheless, in the classic cloud-based/centralized e-health paradigms, all the data will be centrally stored on the server to facilitate model training, which inevitably incurs privacy concerns and high time delay. Distributed solutions like Decentralized Stochastic Gradient Descent (D-SGD) are proposed to provide safe and timely diagnostic…
On the Tradeoff between Privacy Preservation and Byzantine-Robustness in Decentralized Learning
School of Computer Science and Engineering, Sun Yat-Sen University（中山大学计算机科学与工程学院）；Department of Electrical and Computer Engineering, University of California, San Diego（加利福尼亚大学圣地亚哥分校电气与计算机工程系）
Distributed Optimization with Streaming Data: A Temporal Weighting Perspective
School of Electrical, Computer and Energy Engineering, Arizona State University（亚利桑那州立大学电气、计算机与能源工程学院）；Department of Electrical Engineering (ISY), Linköping University（林雪平大学电气工程系）
Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration
College of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间科学与技术学院）
We study stochastic decentralized optimization for the problem of training machine learning models with large-scale distributed data. We extend the widely used EXTRA and DIGing methods with variance reduction (VR), and propose two methods: VR-EXTRA and VR-DIGing. The proposed VR-EXTRA requires the time of $O((κ_s+n)\log\frac{1}ε)$ stochastic gradient evaluations and $O((κ_b+κ_c)\log\frac{1}ε)$ communication rounds to reach precision $ε$, which are the best complexities among the non-accelerated gradient-type methods, where $κ_s$ and $κ_b$ are the stochastic condition number and batch condition…
LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates
Center for Cognitive Interaction Technology, Universität Bielefeld, Germany（认知交互技术中心，比勒菲尔德大学，德国）；Bielefeld University（比勒菲尔德大学）；Department of Computer Science, Royal Holloway, University of London, United Kingdom（计算机科学系，伦敦大学皇家霍洛威学院，英国）
Iowa State University（爱荷华州立大学）；Johnson Controls（江森自控）；New York University（纽约大学）
We consider the problem of decentralized deep learning where multiple agents collaborate to learn from a distributed dataset. While there exist several decentralized deep learning approaches, the majority consider a central parameter-server topology for aggregating the model parameters from the agents. However, such a topology may be inapplicable in networked systems such as ad-hoc mobile networks, field robotics, and power network systems where direct communication with the central parameter server may be inefficient. In this context, we propose and analyze a novel decentralized deep learning…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Google Inc.（谷歌公司）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）
Decentralized algorithm is a form of computation that achieves a global goal through local dynamics that relies on low-cost communication between directly-connected agents. On large-scale optimization tasks involving distributed datasets, decentralized algorithms have shown strong, sometimes superior, performance over distributed algorithms with a central node. Recently, developing decentralized algorithms for deep learning has attracted great attention. They are considered as low-communication-overhead alternatives to those using a parameter server or the Ring-Allreduce protocol. However, the…
LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM T. J. Watson Research Center（IBM T. J. Watson 研究中心）；MIT-IBM Watson AI Lab, IBM Research（MIT-IBM Watson AI 实验室，IBM 研究院）
In this paper, we consider the Byzantine-robust stochastic optimization problem defined over decentralized static and time-varying networks, where the agents collaboratively minimize the summation of expectations of stochastic local cost functions, but some of the agents are unreliable due to data corruptions, equipment failures or cyber-attacks. The unreliable agents, which are called as Byzantine agents thereafter, can send faulty values to their neighbors and bias the optimization process. Our key idea to handle the Byzantine attacks is to formulate a total variation (TV) norm-penalized app…
Provably Accelerated Decentralized Gradient Method Over Unbalanced Directed Graphs
Shanghai Center for Mathematical Sciences, Fudan University（复旦大学上海数学中心）；School of Mathematical Sciences, Fudan University（复旦大学数学科学学院）；Shanghai Key Laboratory for Contemporary Applied Mathematics（上海现代应用数学重点实验室）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳）数据科学学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Recent developments in machine learning have shown that successful models do not rely only on huge amounts of data but the right kind of data. We show in this paper how this data-centric approach can be facilitated in a decentralized manner to enable efficient data collection for algorithms. Face detectors are a class of models that suffer heavily from bias issues as they have to work on a large variety of different data. We also propose a face detection and anonymization approach using a hybrid MultiTask Cascaded CNN with FaceNet Embeddings to benchmark multiple datasets to describe and evalu…
High-Probability Nash Regret for Decentralized Learning in Markov $α$-Potential Games: Episodic and Fully Online Asynchronous Algorithms with Applications to Markov Congestion Games
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
The convergence speed of machine learning models trained with Federated Learning is significantly affected by heterogeneous data partitions, even more so in a fully decentralized setting without a central server. In this paper, we show that the impact of label distribution skew, an important type of data heterogeneity, can be significantly reduced by carefully designing the underlying communication topology. We present D-Cliques, a novel topology that reduces gradient bias by grouping nodes in sparsely interconnected cliques such that the label distribution in a clique is representative of the…
iSING Lab, The Hong Kong University of Science and Technology（香港科技大学iSING实验室）；Clustar；Peng Cheng Lab（鹏城实验室）
In recent years, federated learning (FL) has been widely applied for supporting decentralized collaborative learning scenarios. Among existing FL models, federated logistic regression (FLR) is a widely used statistic model and has been used in various industries. To ensure data security and user privacy, FLR leverages homomorphic encryption (HE) to protect the exchanged data among different collaborative parties. However, HE introduces significant computational overhead (i.e., the cost of data encryption/decryption and calculation over encrypted data), which eventually becomes the performance …
Inner Mongolia University of Technology（内蒙古工业大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；New Jersey Institute of Technology（新泽西理工学院）
Advanced researches on connected vehicles have recently targeted to the integration of vehicle-to-everything (V2X) networks with Machine Learning (ML) tools and distributed decision making. Federated learning (FL) is emerging as a new paradigm to train machine learning (ML) models in distributed systems, including vehicles in V2X networks. Rather than sharing and uploading the training data to the server, the updating of model parameters (e.g., neural networks' weights and biases) is applied by large populations of interconnected vehicles, acting as local learners. Despite these benefits, the …
Decentralized Machine Learning with Centralized Performance Guarantees via Gibbs Algorithms
1 Centre Inria d'Université Côte d'Azur, INRIA, Sophia Antipolis, France.；2 Laboratoire GAATI, Universit\' e de la Polyn\' e sie fran c aise, Fa`a`\= a , French Polynesia.；3 School of Electrical；Electronic Engineering, University of Sheffield, Sheffield, United Kingdom.；4 ECE Dept. Princeton University, Princeton, 08544 NJ, USA.
Non-Expansive Two-Time-Scale Stochastic Approximation: A Fixed-Schedule One-Quarter Barrier and Bias-Corrected Acceleration
Indian Institute of Technology Kharagpur（印度理工学院Khargapur分校）；Mohamed bin Zayed University of Artificial Intelligence（Mohamed bin Zayed人工智能大学）；Purdue University（普渡大学）
F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；Tongji University（同济大学）；Shanghai Research Institute for Intelligent Autonomous Systems（上海智能自主系统研究院）；East China Normal University（华东师范大学）；Shanghai Jiao Tong University（上海交通大学）
Peking University（北京大学）；Chinese Academy of Sciences（中国科学院）；University College London（伦敦大学学院）；King’s College London（伦敦国王学院）；Xiamen University（厦门大学）；BIGAI（北京通用人工智能研究院）
Reinforcement learning algorithms require a large amount of samples; this often limits their real-world applications on even simple tasks. Such a challenge is more outstanding in multi-agent tasks, as each step of operation is more costly requiring communications or shifting or resources. This work aims to improve data efficiency of multi-agent control by model-based learning. We consider networked systems where agents are cooperative and communicate only locally with their neighbors, and propose the decentralized model-based policy optimization framework (DMPO). In our method, each agent lear…
Duke University（杜克大学）；Hong Kong University of Science and Technology（香港科技大学）；Texas A&M University（得克萨斯农工大学）
Applying machine learning (ML) in design flow is a popular trend in EDA with various applications from design quality predictions to optimizations. Despite its promise, which has been demonstrated in both academic researches and industrial tools, its effectiveness largely hinges on the availability of a large amount of high-quality training data. In reality, EDA developers have very limited access to the latest design data, which is owned by design companies and mostly confidential. Although one can commission ML model training to a design company, the data of a single company might be still i…
