参数服务器采用编码梯度的离散鲁棒分布式Optimization
Straggler-Robust Distributed Optimization with the Parameter Server Utilizing Coded Gradient
摘要：
分布式网络中的优化在几乎所有分布式机器学习问题中起着核心作用。原则上，分布式任务分配的使用减少了计算时间，允许更好的响应率和更高的数据可靠性。然而，为了使这些计算算法在复杂的分布式系统中有效地运行，这些算法应该补偿通信冗余、网络节点故障和延迟（称为掉队者）。这些问题可能会改变网络的有效连接拓扑，这可能会随着时间的推移而变化，从而阻碍优化过程。本文提出了一种新的分布式无约束优化算法，用于求解强凸函数的极小化问题，该算法适用于参数服务器网络。特别地，网络工作节点解决它们的局部最优化问题，允许计算它们的局部编码梯度，并将它们发送到不同的服务器节点。然后，每个服务器节点聚合其传递的局部梯度，允许收敛到所需的优化器。该算法对网络工作节点故障或断开连接，或被称为掉队者的延迟具有鲁棒性。克服落伍者问题的一种方法是允许在网络上编码。我们进一步扩展了这个编码框架，以提高这种不同的网络拓扑结构下的算法的收敛性。最后，我们在MATLAB中实现了所提出的方案，并提供了比较结果，证明了所提出的框架的有效性。
英文摘要：
Optimization in distributed networks plays a central role in almost all distributed machine learning problems. In principle, the use of distributed task allocation has reduced the computational time, allowing better response rates and higher data reliability. However, for these computational algorithms to run effectively in complex distributed systems, the algorithms ought to compensate for communication asynchrony, and network node failures and delays known as stragglers. These issues can change the effective connection topology of the network, which may vary through time, thus hindering the optimization process. In this paper, we propose a new distributed unconstrained optimization algorithm for minimizing a strongly convex function which is adaptable to a parameter server network. In particular, the network worker nodes solve their local optimization problems, allowing the computation of their local coded gradients, and send them to different server nodes. Then each server node aggregates its communicated local gradients, allowing convergence to the desired optimizer. This algorithm is robust to network worker node failures or disconnection, or delays known as stragglers. One way to overcome the straggler problem is to allow coding over the network. We further extend this coding framework to enhance the convergence of the proposed algorithm under such varying network topologies. Finally, we implement the proposed scheme in MATLAB and provide comparative results demonstrating the effectiveness of the proposed framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
