arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.12403cs.LGcs.DC
收藏

图神经网络中的协作式小批量处理

Cooperative Minibatching in Graph Neural Networks

School of Computational Science and Engineering(计算科学与工程学院);Georgia Institute of Technology(佐治亚理工学院);NVIDIA Corporation(英伟达公司)

Muhammed Fatih Balin, Dominique LaSalle, Ümit V. Çatalyürek

AI总结:

针对GNN小批量训练中跨处理单元的重复计算与邻域爆炸问题,提出协作式小批量及依赖连续批量方法,显著节省带宽并提升训练速度。

摘要:

训练大规模图神经网络(GNN)需要大量计算资源,且该过程具有高度数据密集性。降低资源需求最有效的方法之一是将小批量训练与图采样相结合。GNN具有一个独特性质:一个小批量中的各个数据项存在重叠数据。然而,通常实现的独立小批量处理方法会为每个处理单元(PE,即核心和/或GPU)分配各自的小批量进行处理,从而导致PE之间出现重复计算和输入数据访问。这会放大邻域爆炸现象(NEP),而该现象正是限制扩展的主要瓶颈。为在多PE设置中减轻NEP的影响,我们提出一种称为协作式小批量处理的新方法。我们的方法利用了这样一个事实:采样子图的大小是批量大小的凹函数,因此随着批量大小增加,工作量会显著减少。所以,即使全局批量大小相同,对于配备快速互连的处理器而言,像单个更大的处理器一样共同处理一个大的小批量,也要比分别处理多个较小的小批量更有利。我们还展示了如何通过生成具有依赖关系的连续小批量,在串行执行中利用同一现象。实验评估表明,只需增加这种依赖关系,即可在获取顶点嵌入时节省最高4倍的带宽,且不会损害模型收敛。结合我们提出的这些方法,在使用相同资源的情况下,单节点多GPU系统上相比独立小批量处理实现了最高64%的加速。

英文摘要:

Training large scale Graph Neural Networks (GNNs) requires significant computational resources, and the process is highly data-intensive. One of the most effective ways to reduce resource requirements is minibatch training coupled with graph sampling. GNNs have the unique property that items in a minibatch have overlapping data. However, the commonly implemented Independent Minibatching approach assigns each Processing Element (PE, i.e., cores and/or GPUs) its own minibatch to process, leading to duplicated computations and input data access across PEs. This amplifies the Neighborhood Explosion Phenomenon (NEP), which is the main bottleneck limiting scaling. To reduce the effects of NEP in the multi-PE setting, we propose a new approach called Cooperative Minibatching. Our approach capitalizes on the fact that the size of the sampled subgraph is a concave function of the batch size, leading to significant reductions in the amount of work as batch sizes increase. Hence, it is favorable for processors equipped with a fast interconnect to work on a large minibatch together as a single larger processor, instead of working on separate smaller minibatches, even though global batch size is identical. We also show how to take advantage of the same phenomenon in serial execution by generating dependent consecutive minibatches. Our experimental evaluations show up to 4x bandwidth savings for fetching vertex embeddings, by simply increasing this dependency without harming model convergence. Combining our proposed approaches, we achieve up to 64% speedup over Independent Minibatching on single-node multi-GPU systems, using same resources.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑