参数服务器网络中的离散鲁棒分布式Optimization
Straggler-Robust Distributed Optimization in Parameter-Server Networks
摘要：
分布式网络中的优化在几乎所有分布式机器学习问题中起着核心作用。原则上，分布式任务分配的使用减少了计算时间，允许更好的响应率和更高的数据可靠性。然而，为了使这些计算算法在复杂的分布式系统中有效地运行，这些算法应该补偿被称为掉队者的通信延迟、网络节点故障和延迟。这些问题可能会改变网络的有效连接拓扑，这可能会随着时间的推移而变化，从而阻碍优化过程。本文提出了一种新的分布式无约束优化算法，该算法适用于参数服务器网络中的凸函数最小化问题。特别地，网络工作节点解决它们的局部最优化问题，允许计算它们的局部编码梯度，其将被发送到不同的服务器节点。然后，在该参数服务器平台内，每个服务器节点聚合其通信的局部梯度，允许收敛到期望的优化器。该算法对网络中的工作节点失效、断开连接或延迟节点（称为掉队者）具有鲁棒性。克服落伍者问题的一种方法是允许在网络上编码。我们进一步扩展了这个编码框架，以提高这种不同的网络拓扑结构下的算法的收敛性。通过使用编码和利用一致有界延迟的梯度的评估，我们进一步提高了所提出的算法的性能。最后，我们在MATLAB中实现了该方案，并提供了比较结果，证明了该框架的有效性
英文摘要：
Optimization in distributed networks plays a central role in almost all distributed machine learning problems. In principle, the use of distributed task allocation has reduced the computational time, allowing better response rates and higher data reliability. However, for these computational algorithms to run effectively in complex distributed systems, the algorithms ought to compensate for communication asynchrony, network node failures and delays known as stragglers. These issues can change the effective connection topology of the network, which may vary over time, thus hindering the optimization process. In this paper, we propose a new distributed unconstrained optimization algorithm for minimizing a convex function which is adaptable to a parameter server network. In particular, the network worker nodes solve their local optimization problems, allowing the computation of their local coded gradients, which will be sent to different server nodes. Then within this parameter server platform each server node aggregates its communicated local gradients, allowing convergence to the desired optimizer. This algorithm is robust to network s worker node failures, disconnection, or delaying nodes known as stragglers. One way to overcome the straggler problem is to allow coding over the network. We further extend this coding framework to enhance the convergence of the proposed algorithm under such varying network topologies. By using coding and utilizing evaluations of gradients of uniformly bounded delay we further enhance the proposed algorithm performance. Finally, we implement the proposed scheme in MATLAB and provide comparative results demonstrating the effectiveness of the proposed framework
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
