GRE:一个面向大规模分布式图并行应用的图搜索引擎
GRE: A Graph Runtime Engine for Large-Scale Distributed Graph-Parallel Applications
摘要:
大规模分布式图并行计算是一个挑战。一方面,由于计算模式的不规则性和局部性的缺乏,很难有效地表示并行性。另一方面,由于无标度性质,现实世界的图很难在低割下平衡划分。为了应对这些挑战,最近开发了几个图形并行框架,包括Pregel和GraphLab(PowerGraph)。在本文中,我们提出了一个替代框架,图形搜索引擎(GRE)。GRE在保留顶点中心编程模型的基础上,提出了两个新的抽象:1)基于主动消息的Scatter-Combine计算模型,以实现大规模的细粒度边级并行; 2)基于顶点分解的Agent-Graph数据模型,以划分和表示有向图。GRE是在商用现成的多核集群上实现的。我们实验评估GRE与三个基准程序(PageRank,单源最短路径和连接组件)在现实世界和合成图的亿万个顶点。与PowerGraph相比,GRE在8~16台机器(192核)上的性能提高了2.5~17倍。具体来说,GRE中的PageRank与公开文献中报道的其他框架(PowerGraph,Spark,Twister)相比是最快的。此外,GRE显著地优化了内存使用,因此它可以在我们的集群上处理10亿个顶点和170亿条边的大型图形,总共768GB内存,而PowerGraph只能处理不到一半的图形规模。
英文摘要:
Large-scale distributed graph-parallel computing is challenging. On one hand, due to the irregular computation pattern and lack of locality, it is hard to express parallelism efficiently. On the other hand, due to the scale-free nature, real-world graphs are hard to partition in balance with low cut. To address these challenges, several graph-parallel frameworks including Pregel and GraphLab (PowerGraph) have been developed recently. In this paper, we present an alternative framework, Graph Runtime Engine (GRE). While retaining the vertex-centric programming model, GRE proposes two new abstractions: 1) a Scatter-Combine computation model based on active message to exploit massive fined-grained edge-level parallelism, and 2) a Agent-Graph data model based on vertex factorization to partition and represent directed graphs. GRE is implemented on commercial off-the-shelf multi-core cluster. We experimentally evaluate GRE with three benchmark programs (PageRank, Single Source Shortest Path and Connected Components) on real-world and synthetic graphs of millions billion of vertices. Compared to PowerGraph, GRE shows 2.5~17 times better performance on 8~16 machines (192 cores). Specifically, the PageRank in GRE is the fastest when comparing to counterparts of other frameworks (PowerGraph, Spark,Twister) reported in public literatures. Besides, GRE significantly optimizes memory usage so that it can process a large graph of 1 billion vertices and 17 billion edges on our cluster with totally 768GB memory, while PowerGraph can only process less than half of this graph scale.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。