arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1602.01421cs.DCcs.MS
收藏

基于SSD的十亿节点图谱分析特征解算器

An SSD-based eigensolver for spectral analysis on billion-node graphs

Da Zheng, Randal Burns, Joshua Vogelstein, Carey E. Priebe, Alexander S. Szalay

摘要:

许多特征值求解器,如ARPACK和Anasazi已被开发用于计算大型稀疏矩阵的特征值。这些特征值求解器受到RAM容量的限制。对于较小的特征值问题,它们在单个机器的内存中运行,而对于较大的问题,它们需要分布式内存。相比之下,我们开发了一个基于SSD的特征值求解器框架,称为FlashEigen,它将Anasazi特征值求解器扩展到SSD,以在一台机器中计算具有数亿甚至数十亿顶点的图的特征值。FlashEigen以半外部存储器方式执行稀疏矩阵乘法,即,我们将稀疏矩阵保存在SSD上,将密集矩阵保存在内存中。我们将整个向量子空间存储在SSD上,并通过缓存最新的密集矩阵来减少I/O以提高性能。我们的研究结果表明,FlashEigen能够实现40%-60%的性能,其在内存中的实现,并具有性能媲美的Anasazi eigensolvers上的机器与48个CPU内核。此外,它能够扩展到具有34亿个顶点和1290亿条边的图形。使用120 GB内存计算十亿节点图的八个特征值大约需要四个小时。

英文摘要:

Many eigensolvers such as ARPACK and Anasazi have been developed to compute eigenvalues of a large sparse matrix. These eigensolvers are limited by the capacity of RAM. They run in memory of a single machine for smaller eigenvalue problems and require the distributed memory for larger problems. In contrast, we develop an SSD-based eigensolver framework called FlashEigen, which extends Anasazi eigensolvers to SSDs, to compute eigenvalues of a graph with hundreds of millions or even billions of vertices in a single machine. FlashEigen performs sparse matrix multiplication in a semi-external memory fashion, i.e., we keep the sparse matrix on SSDs and the dense matrix in memory. We store the entire vector subspace on SSDs and reduce I/O to improve performance through caching the most recent dense matrix. Our result shows that FlashEigen is able to achieve 40%-60% performance of its in-memory implementation and has performance comparable to the Anasazi eigensolvers on a machine with 48 CPU cores. Furthermore, it is capable of scaling to a graph with 3.4 billion vertices and 129 billion edges. It takes about four hours to compute eight eigenvalues of the billion-node graph using 120 GB memory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑