arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.07482cs.LG
收藏

FreshGNN:通过稳定的历史嵌入减少图神经网络训练中的内存访问

FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Training

Tsinghua University(清华大学);New York University(纽约大学);Amazon(亚马逊公司);MIT(麻省理工学院);Fudan University(复旦大学)

Kezhao Huang, Haitian Jiang, Minjie Wang, Guangxuan Xiao, David Wipf, Xiang Song, Quan Gan, Zengfeng Huang, Jidong Zhai, Zheng Zhang

AI总结:

FreshGNN通过历史缓存存储和重用GNN节点嵌入,结合基于梯度和陈旧度的缓存策略,显著减少内存访问并加速大型图数据集上的GNN训练,同时保持测试精度。

摘要:

在大型真实图上训练图神经网络(GNN)模型时,一个关键的性能瓶颈是将节点特征加载到GPU上。由于GPU内存有限,必须进行昂贵的数据移动,以便将这些特征存储在访问速度较慢的替代设备(如CPU内存)上。此外,图结构的不规则性导致数据局部性差,进一步加剧了这一问题。因此,能够高效训练大型GNN模型的现有框架通常会因当前可用的捷径而导致精度显著下降。为了解决这些限制,我们提出了FreshGNN,这是一个通用的GNN小批量训练框架,它利用历史缓存来存储和重用GNN节点嵌入,而不是在每次迭代中通过获取原始特征来重新计算它们。其成功的关键在于,相应的缓存策略结合了基于梯度和陈旧度的标准,从需要重新计算以减少估计误差和后续下游精度损失的嵌入中,选择性地筛选出相对稳定且可缓存的嵌入。配合支持这种选择性历史缓存的互补系统增强,FreshGNN能够在ogbn-papers100M和MAG240M等大型图数据集上将训练速度提高3.4倍至20.5倍,并减少59%的内存访问,对测试精度的影响不到1%。

英文摘要:

A key performance bottleneck when training graph neural network (GNN) models on large, real-world graphs is loading node features onto a GPU. Due to limited GPU memory, expensive data movement is necessary to facilitate the storage of these features on alternative devices with slower access (e.g. CPU memory). Moreover, the irregularity of graph structures contributes to poor data locality which further exacerbates the problem. Consequently, existing frameworks capable of efficiently training large GNN models usually incur a significant accuracy degradation because of the currently-available shortcuts involved. To address these limitations, we instead propose FreshGNN, a general-purpose GNN mini-batch training framework that leverages a historical cache for storing and reusing GNN node embeddings instead of re-computing them through fetching raw features at every iteration. Critical to its success, the corresponding cache policy is designed, using a combination of gradient-based and staleness criteria, to selectively screen those embeddings which are relatively stable and can be cached, from those that need to be re-computed to reduce estimation errors and subsequent downstream accuracy loss. When paired with complementary system enhancements to support this selective historical cache, FreshGNN is able to accelerate the training speed on large graph datasets such as ogbn-papers100M and MAG240M by 3.4x up to 20.5x and reduce the memory access by 59%, with less than 1% influence on test accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑