探索分布式图素描的图景
Exploring the Landscape of Distributed Graph Sketching
AI总结:
本文提出分布式图流处理系统Landscape,利用线性素描技术将计算分配给无需通信的工作节点,克服了CPU和网络瓶颈,实现了高吞吐量与低查询延迟。
摘要:
近期的研究开始使用图素描(graph sketching)方法进行稠密图处理,该方法通过对输入图的信息进行有损压缩来大幅降低空间开销。在本文中,我们探索了素描算法奇特且令人惊讶的性能图景。我们重点强调了它们在处理以前研究成本过高的稠密图时令人惊讶的优势,以及该技术目前的局限性。最值得注意的是,我们展示了素描如何避免限制传统图处理方法的瓶颈。单机流式图处理系统通常受限于CPU性能瓶颈,而分布式图处理系统通常受限于网络延迟瓶颈。我们提出了Landscape,这是一个使用线性素描(linear sketching)的分布式图流处理系统,它将计算图属性的CPU工作分配给分布式工作节点,而无需工作节点之间的通信。因此,它克服了限制其他系统的CPU和网络瓶颈。事实上,对于连通分量(connected components)问题,Landscape实现了相当于最大持续RAM带宽四分之一的流摄取速率,并且比随机访问RAM带宽快四倍。此外,我们证明,对于任何图更新和查询序列,Landscape消耗的网络带宽最多比接收输入流所需的网络带宽多出一个常数因子。我们展示了该系统在具有$2^{17}$个顶点的图上每秒可以摄取高达3.32亿次流更新。我们展示了它随更多分布式计算能力具有良好的可扩展性:在由40台分布式工作机器组成的集群中,其更新摄取速度是1台分布式工作机器的35倍。Landscape使用启发式方法将其查询延迟比先前最先进技术降低了多达四个数量级。
英文摘要:
Recent work has initiated the study of dense graph processing using graph sketching methods, which drastically reduce space costs by lossily compressing information about the input graph. In this paper, we explore the strange and surprising performance landscape of sketching algorithms. We highlight both their surprising advantages for processing dense graphs that were previously prohibitively expensive to study, as well as the current limitations of the technique. Most notably, we show how sketching can avoid bottlenecks that limit conventional graph processing methods. Single-machine streaming graph processing systems are typically bottlenecked by CPU performance, and distributed graph processing systems are typically bottlenecked by network latency. We present Landscape, a distributed graph-stream processing system that uses linear sketching to distribute the CPU work of computing graph properties to distributed workers with no need for worker-to-worker communication. As a result, it overcomes the CPU and network bottlenecks that limit other systems. In fact, for the connected components problem, Landscape achieves a stream ingestion rate one-fourth that of maximum sustained RAM bandwidth, and is four times faster than random access RAM bandwidth. Additionally, we prove that for any sequence of graph updates and queries Landscape consumes at most a constant factor more network bandwidth than is required to receive the input stream. We show that this system can ingest up to 332 million stream updates per second on a graph with $2^{17}$ vertices. We show that it scales well with more distributed compute power: given a cluster of 40 distributed worker machines, it can ingest updates 35 times as fast as with 1 distributed worker machine. Landscape uses heuristics to reduce its query latency by up to four orders of magnitude over the prior state of the art.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。