arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.00837cs.DC
收藏

Helios:一种在 TB 级图上实现内存级性能的高效核外 GNN 训练系统

Helios: An Efficient Out-of-core GNN Training System on Terabyte-scale Graphs with In-memory Performance

Jie Sun, Mo Sun, Zheng Zhang, Jun Xie, Zuocheng Shi, Zihan Yang, Jie Zhang, Fei Wu, Zeke Wang

AI总结:

针对单机 TB 级图 GNN 训练的磁盘 IO 与 GPU 利用率瓶颈,本文提出 Helios,通过 GPU 发起异步 IO、GPU 管理异构缓存和 GNN 感知深度流水线,达到内存系统级吞吐量。

摘要:

在大规模图数据上训练图神经网络(GNN)对众多实际应用具有巨大潜力,但仍是一项重大挑战。已有若干基于磁盘的 GNN 系统被构建出来,用于在单机上训练大规模图。然而,它们在性能方面往往存在不足,尤其是在 TB 级图上训练时。这是因为现有基于磁盘的系统要么过度关注最小化 SSD 访问次数,要么未能使 SSD 访问与 GNN 训练充分重叠,从而在 CPU 端产生大量不必要的开销,进而导致 GPU 利用率较低。为此,我们提出 Helios,该系统能够在单机上训练 TB 级图的 GNN,同时实现可与内存系统相媲美的吞吐量。为实现这一点,我们首先提出一种由 GPU 发起的异步磁盘 IO 栈,使 GPU 能够直接访问 SSD 上的图数据。该设计仅需约 30% 的 GPU 核心即可达到几乎最大的磁盘 IO 吞吐量,并且在 IO 提交与 IO 完成之间不浪费 GPU 核心,从而将大多数 GPU 核心留给其他 GNN 内核。其次,我们设计了由 GPU 管理的异构缓存,将缓存层次扩展到异构的 CPU 和 GPU 内存,并因此通过 GPU 并行显著提高缓存查找吞吐量。最后,我们构建了深度感知 GNN 的流水线,将整个 GNN 训练过程的计算与通信阶段无缝集成,最大化 GPU 计算周期的利用。实验结果表明,即使对于 TB 级图,Helios 也能匹敌内存 GNN 系统的训练吞吐量。值得注意的是,在所有 TB 级图上,Helios 相比最先进的 GPU 管理基线最高提升 6.43 倍,并超过 CPU 管理基线 182 倍以上。

英文摘要:

Training graph neural networks (GNNs) on large-scale graph data holds immense promise for numerous real-world applications but remains a great challenge. Several disk-based GNN systems have been built to train large-scale graphs in a single machine. However, they often fall short in terms of performance, especially when training on terabyte-scale graphs. This is because existing disk-based systems either overly focus on minimizing the number of SSD accesses or do not fully overlap SSD accesses with GNN training, thus resulting in substantial unnecessary overhead on the CPU side and then low GPU utilization. To this end, we propose Helios, a system that can train GNN on terabyte graphs in a single machine while achieving throughput comparable with in-memory systems. To achieve this, we first present a GPU-initiated asynchronous disk IO stack, allowing the GPU to directly access graph data on SSD. This design only requires about 30% GPU cores to reach the almost maximal disk IO throughput and wastes no GPU cores between IO submission and IO completion such that the majority of GPU cores are left for other GNN kernels. Second, we design a GPU-managed heterogeneous cache that extends the cache hierarchy to heterogeneous CPU and GPU memory and thus enhances cache lookup throughput significantly by GPU parallelism. Finally, we build a deep GNN-aware pipeline that seamlessly integrates the computation and communication phases of the entire GNN training process, maximizing the utility of GPU computation cycles. Experimental results demonstrate that Helios can match the training throughput of in-memory GNN systems, even for terabyte-scale graphs. Remarkably, Helios surpasses the state-of-the-art GPU-managed baselines by up to 6.43x and exceeds CPU-managed baselines by over 182x on all terabyte-scale graphs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑