arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.03152cs.LGcs.DCcs.PF
收藏

结合概率邻域扩展分析与缓存的通信高效图神经网络

Communication-Efficient Graph Neural Networks with Probabilistic Neighborhood Expansion Analysis and Caching

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);IBM Research(IBM研究院)

Tim Kaler, Alexandros-Stavros Iliopoulos, Philip Murzynowski, Tao B. Schardl, Charles E. Leiserson, Jie Chen

AI总结:

针对分布式GNN训练的特征通信瓶颈,本文提出基于顶点包含概率的缓存策略,构建SALIENT++系统,在大幅降低通信量与存储开销的同时显著提升训练速度。

摘要:

自图神经网络(GNN)诞生以来,鉴于其在推荐系统、金融取证等应用中的广泛应用与成功,研究者们一直在积极研究大规模图上的GNN训练与推理。本文聚焦分布式环境下采用逐节点采样的GNN小批量训练与推理场景:由于顶点特征需跨分布式存储分区存储,特征通信成为制约可扩展性的主要瓶颈。为在不损失预测精度的前提下大幅降低通信量,我们提出一种缓存策略,用于缓存远程分区中频繁访问顶点的相关数据。该策略基于多跳邻域采样过程中的顶点包含概率(VIP)分析——多跳采样可能使邻域范围远超图的分区边界。VIP分析不仅能消除通信瓶颈,还提供了一种内存数据组织方式,即为访问最频繁的顶点特征优先分配GPU存储空间。我们提出SALIENT++系统,它对现有先进系统SALIENT进行扩展,使其支持分区特征数据,并采用VIP驱动的缓存策略。SALIENT++通过深度流水线保留了SALIENT的本地训练效率与可扩展性,同时大幅降低通信量,且仅需SALIENT所需存储的一小部分。我们在Open Graph Benchmark数据集上开展了实验,结果表明:在8台单GPU机器上用SALIENT++训练3层GraphSAGE模型,速度是1台单GPU机器上SALIENT的7.1倍,是8台单GPU机器上DistDGL的12.7倍。

英文摘要:

Training and inference with graph neural networks (GNNs) on massive graphs has been actively studied since the inception of GNNs, owing to the widespread use and success of GNNs in applications such as recommendation systems and financial forensics. This paper is concerned with minibatch training and inference with GNNs that employ node-wise sampling in distributed settings, where the necessary partitioning of vertex features across distributed storage causes feature communication to become a major bottleneck that hampers scalability. To significantly reduce the communication volume without compromising prediction accuracy, we propose a policy for caching data associated with frequently accessed vertices in remote partitions. The proposed policy is based on an analysis of vertex-wise inclusion probabilities (VIP) during multi-hop neighborhood sampling, which may expand the neighborhood far beyond the partition boundaries of the graph. VIP analysis not only enables the elimination of the communication bottleneck, but it also offers a means to organize in-memory data by prioritizing GPU storage for the most frequently accessed vertex features. We present SALIENT++, which extends the prior state-of-the-art SALIENT system to work with partitioned feature data and leverages the VIP-driven caching policy. SALIENT++ retains the local training efficiency and scalability of SALIENT by using a deep pipeline and drastically reducing communication volume while consuming only a fraction of the storage required by SALIENT. We provide experimental results with the Open Graph Benchmark data sets and demonstrate that training a 3-layer GraphSAGE model with SALIENT++ on 8 single-GPU machines is 7.1 faster than with SALIENT on 1 single-GPU machine, and 12.7 faster than with DistDGL on 8 single-GPU machines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑