小集群上基于外部存储的大规模社交网络分布式生成
External Memory based Distributed Generation of Massive Scale Social Networks on Small Clusters
摘要：
小型分布式系统受其主存储器的限制，无法生成超大型图形。对当前图形生成器进行简单扩展以利用外部存储器会导致大量的随机I/O，因此不会随大小而扩展。在这项工作中，我们提供了一种技术来生成大规模的图形上的小集群的计算节点有限的主内存。我们开发了几个分布式和外部存储器算法，主要是，洗牌，重新标记，重新分配，和压缩稀疏行（csr）转换。这些算法在MPI/pthread模型中实现，以帮助在每个核内的多核之间并行化操作。使用我们的方案，它是可行的，以产生一个图形的大小$2^{38}$节点（规模38），只使用64个计算节点。这可以与目前的方案相比，假设64 GB的主内存，至少需要8192个计算节点。我们的工作对外部存储器图形库（如STXXL）和基于SSD的超级计算机（如Dash和Gordon）上的图形处理具有更广泛的影响[1][2]。
英文摘要：
Small distributed systems are limited by their main memory to generate massively large graphs. Trivial extension to current graph generators to utilize external memory leads to large amount of random I/O hence do not scale with size. In this work we offer a technique to generate massive scale graphs on small cluster of compute nodes with limited main memory. We develop several distributed and external memory algorithms, primarily, shuffle, relabel, redistribute, and, compressed-sparse-row (csr) convert. The algorithms are implemented in MPI/pthread model to help parallelize the operations across multicores within each core. Using our scheme it is feasible to generate a graph of size $2^{38}$ nodes (scale 38) using only 64 compute nodes. This can be compared with the current scheme would require at least 8192 compute node, assuming 64GB of main memory. Our work has broader implications for external memory graph libraries such as STXXL and graph processing on SSD-based supercomputers such as Dash and Gordon [1][2].
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
