arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1402.2394cs.DB
收藏

GraphX:统一数据并行和图形并行分析

GraphX: Unifying Data-Parallel and Graph-Parallel Analytics

Reynold S. Xin, Daniel Crankshaw, Ankur Dave, Joseph E. Gonzalez, Michael J. Franklin, Ion Stoica

摘要:

从社交网络到语言建模,图数据的规模和重要性的增长已经驱动了许多新的图并行系统(例如,Pregel,GraphLab)。通过限制可以表达的计算并引入新技术来划分和分布图,这些系统可以有效地执行迭代图算法,速度比更一般的数据并行系统快几个数量级。然而,实现性能增益的相同限制也使得难以表达典型图分析管道中的许多重要阶段:构建图,修改其结构或表达跨越多个图的计算。因此,现有的图分析管道使用外部存储系统组成图并行和数据并行系统,导致大量的数据移动和复杂的编程模型。为了解决这些挑战,我们引入了GraphX,这是一个分布式图计算框架,它统一了图并行和数据并行计算。GraphX提供了一个小的、核心的图并行运算符集,其表达能力足以实现Pregel和PowerGraph抽象,但又足够简单,可以转换为关系代数。GraphX使用了一系列查询优化技术,如自动连接重写,以有效地实现这些图并行运算符。我们在真实世界的图形和工作负载上评估了GraphX,并证明GraphX实现了与专用图形计算系统相当的性能,同时在端到端图形管道中表现出色。此外,GraphX在表现力、性能和易用性之间实现了平衡。

英文摘要:

From social networks to language modeling, the growing scale and importance of graph data has driven the development of numerous new graph-parallel systems (e.g., Pregel, GraphLab). By restricting the computation that can be expressed and introducing new techniques to partition and distribute the graph, these systems can efficiently execute iterative graph algorithms orders of magnitude faster than more general data-parallel systems. However, the same restrictions that enable the performance gains also make it difficult to express many of the important stages in a typical graph-analytics pipeline: constructing the graph, modifying its structure, or expressing computation that spans multiple graphs. As a consequence, existing graph analytics pipelines compose graph-parallel and data-parallel systems using external storage systems, leading to extensive data movement and complicated programming model. To address these challenges we introduce GraphX, a distributed graph computation framework that unifies graph-parallel and data-parallel computation. GraphX provides a small, core set of graph-parallel operators expressive enough to implement the Pregel and PowerGraph abstractions, yet simple enough to be cast in relational algebra. GraphX uses a collection of query optimization techniques such as automatic join rewrites to efficiently implement these graph-parallel operators. We evaluate GraphX on real-world graphs and workloads and demonstrate that GraphX achieves comparable performance as specialized graph computation systems, while outperforming them in end-to-end graph pipelines. Moreover, GraphX achieves a balance between expressiveness, performance, and ease of use.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑