大规模构建图：Union Find Shuffle
Building Graphs at a Large Scale: Union Find Shuffle
摘要：
使用分布式计算框架的大规模图形处理在工业中变得普遍和高效。在这项工作中，我们提出了一个高度可扩展和可配置的分布式算法，用于构建连接组件，称为联合查找洗牌（UFS）与路径压缩。该算法的规模和复杂度是数据最初被划分成的分区的数量以及连通分量的大小的函数。我们讨论的复杂性和基准相比，类似的方法。我们还介绍了我们的生产系统的当前基准，运行在商品开箱即用的云Hadoop基础设施上，该算法在一年多前部署，扩展到约750亿个节点和600亿个链接（并且还在增长）。我们强调了我们算法的关键方面，即使在存在具有100亿个节点大小的大型连接组件的偏斜数据的情况下，也可以实现无缝扩展和性能。
英文摘要：
Large scale graph processing using distributed computing frameworks is becoming pervasive and efficient in the industry. In this work, we present a highly scalable and configurable distributed algorithm for building connected components, called Union Find Shuffle (UFS) with Path Compression. The scale and complexity of the algorithm are a function of the number of partitions into which the data is initially partitioned, and the size of the connected components. We discuss the complexity and the benchmarks compared to similar approaches. We also present current benchmarks of our production system, running on commodity out-of-the-box cloud Hadoop infrastructure, where the algorithm was deployed over a year ago, scaled to around 75 Billion nodes and 60 Billions linkages (and growing). We highlight the key aspects of our algorithm which enable seamless scaling and performance even in the presence of skewed data with large connected components in the size of 10 Billion nodes each.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
