FlashGraph:在一系列商品SSD上处理十亿节点图形
FlashGraph: Processing Billion-Node Graphs on an Array of Commodity SSDs
摘要:
图分析执行许多随机读取和写入,因此,这些工作负载通常在内存中执行。传统上,分析大型图需要一个机器集群,因此聚合内存超过了图的大小。我们证明了多核服务器可以处理具有数十亿个顶点和数千亿条边的图形,利用商品SSD以最小的性能损失。为此,我们在用户空间SSD文件系统上实现了一个图形处理引擎,该文件系统专为高IOPS和极端并行性而设计。我们的半外部存储器图形引擎称为FlashGraph将顶点状态存储在内存中,并将边列表存储在SSD上。它通过将计算与I/O重叠来隐藏延迟。为了节省I/O带宽,FlashGraph只访问来自SSD的应用程序请求的边缘列表;为了增加I/O吞吐量并减少I/O的CPU开销,它保守地合并I/O请求。这些设计最大限度地提高了具有不同I/O特性的应用程序的性能。FlashGraph公开了一个通用且灵活的以顶点为中心的编程接口,可以表达各种各样的图形算法及其优化。我们证明,FlashGraph在半外部存储器中执行许多算法,其性能高达80%的内存中实现,并显着优于PowerGraph,一个流行的分布式内存图形引擎。
英文摘要:
Graph analysis performs many random reads and writes, thus, these workloads are typically performed in memory. Traditionally, analyzing large graphs requires a cluster of machines so the aggregate memory exceeds the graph size. We demonstrate that a multicore server can process graphs with billions of vertices and hundreds of billions of edges, utilizing commodity SSDs with minimal performance loss. We do so by implementing a graph-processing engine on top of a user-space SSD file system designed for high IOPS and extreme parallelism. Our semi-external memory graph engine called FlashGraph stores vertex state in memory and edge lists on SSDs. It hides latency by overlapping computation with I/O. To save I/O bandwidth, FlashGraph only accesses edge lists requested by applications from SSDs; to increase I/O throughput and reduce CPU overhead for I/O, it conservatively merges I/O requests. These designs maximize performance for applications with different I/O characteristics. FlashGraph exposes a general and flexible vertex-centric programming interface that can express a wide variety of graph algorithms and their optimizations. We demonstrate that FlashGraph in semi-external memory performs many algorithms with performance up to 80% of its in-memory implementation and significantly outperforms PowerGraph, a popular distributed in-memory graph engine.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。