arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1503.00626cs.DC
收藏

分布式图计算中有效的消息缩减和负载均衡技术

Effective Techniques for Message Reduction and Load Balancing in Distributed Graph Computation

Da Yan, James Cheng, Yi Lu, Wilfred Ng

摘要:

如今,诸如在线社交网络和通信网络之类的海量图已经变得普遍。为了有效地分析这样的大型图,已经开发了许多分布式图计算系统。这些系统采用“像顶点一样思考”的编程范式,其中程序在迭代中进行,并且在每次迭代中,顶点彼此交换消息。然而,使用Pregel的简单消息传递机制,由于这些顶点的高度或所使用的算法的逻辑,一些顶点可能比其他顶点发送/接收更多的消息。这形成了通信瓶颈,并导致集群中机器之间的工作负载不平衡。在本文中,我们提出了两个有效的消息减少技术:(1)顶点镜像与消息组合,和(2)一个额外的请求响应API。这些技术不仅减少了通过网络交换的消息的总数,而且限制了任何单个顶点发送/接收的消息的数量。我们从理论上分析了我们的技术的有效性,并在我们的开源Pregel实现Pregel+之上实现它们。我们在各种大型真实图上的实验表明,我们的消息减少技术显着提高分布式图计算的性能。

英文摘要:

Massive graphs, such as online social networks and communication networks, have become common today. To efficiently analyze such large graphs, many distributed graph computing systems have been developed. These systems employ the "think like a vertex" programming paradigm, where a program proceeds in iterations and at each iteration, vertices exchange messages with each other. However, using Pregel's simple message passing mechanism, some vertices may send/receive significantly more messages than others due to either the high degree of these vertices or the logic of the algorithm used. This forms the communication bottleneck and leads to imbalanced workload among machines in the cluster. In this paper, we propose two effective message reduction techniques: (1)vertex mirroring with message combining, and (2)an additional request-respond API. These techniques not only reduce the total number of messages exchanged through the network, but also bound the number of messages sent/received by any single vertex. We theoretically analyze the effectiveness of our techniques, and implement them on top of our open-source Pregel implementation called Pregel+. Our experiments on various large real graphs demonstrate that our message reduction techniques significantly improve the performance of distributed graph computation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑