分布式Optimization中通信与计算的平衡
Balancing Communication and Computation in Distributed Optimization
摘要:
分布式优化方法由于其在各个领域的广泛适用性,近年来受到了极大的关注。分布式最优化方法通常由两个关键部分组成:通信和计算。更具体地说,在分布式算法的每次迭代(或每几次迭代),网络中的每个节点都需要与其相邻节点进行某种形式的信息交换(通信)以及与(子)梯度相关的计算步骤(计算)。仅通过迭代次数来判断算法的标准方法忽略了与每次迭代相关的复杂性。此外,部署分布式方法的各种应用可能更喜欢通信和计算的不同组合。出于这种差异,在这项工作中,我们提出了一个自适应的成本框架,根据不同的应用程序的功能来调整成本措施。我们提出了一个灵活的算法框架,通信和计算步骤被明确分解,使算法定制的各种应用程序。我们将此框架应用于著名的分布式梯度下降(DGD)方法,并表明,由此产生的定制算法,我们称之为DGD $^t$,近-DGD $^t$和近-DGD $^+$,比较有利的基础算法,理论和经验。所提出的NEAR-DGD $^+$算法是一种精确的一阶方法,其中通信和计算步骤是嵌套的,并且当通信步骤的数量自适应地增加时,该方法收敛到最优解。我们测试的性能和说明的灵活性的方法,以及实际的变种,二次函数和分类问题,在机器学习中出现的迭代,梯度评估,通信和建议的成本框架。
英文摘要:
Methods for distributed optimization have received significant attention in recent years owing to their wide applicability in various domains. A distributed optimization method typically consists of two key components: communication and computation. More specifically, at every iteration (or every several iterations) of a distributed algorithm, each node in the network requires some form of information exchange with its neighboring nodes (communication) and the computation step related to a (sub)-gradient (computation). The standard way of judging an algorithm via only the number of iterations overlooks the complexity associated with each iteration. Moreover, various applications deploying distributed methods may prefer a different composition of communication and computation. Motivated by this discrepancy, in this work we propose an adaptive cost framework which adjusts the cost measure depending on the features of various applications. We present a flexible algorithmic framework, where communication and computation steps are explicitly decomposed to enable algorithm customization for various applications. We apply this framework to the well-known distributed gradient descent (DGD) method, and show that the resulting customized algorithms, which we call DGD$^t$, NEAR-DGD$^t$ and NEAR-DGD$^+$, compare favorably to their base algorithms, both theoretically and empirically. The proposed NEAR-DGD$^+$ algorithm is an exact first-order method where the communication and computation steps are nested, and when the number of communication steps is adaptively increased, the method converges to the optimal solution. We test the performance and illustrate the flexibility of the methods, as well as practical variants, on quadratic functions and classification problems that arise in machine learning, in terms of iterations, gradient evaluations, communications and the proposed cost framework.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。