Graph neural networks (GNNs) are powerful tools for learning from graph data and are widely used in various applications such as social network recommendation, fraud detection, and graph search. The graphs in these applications are typically large, usually containing hundreds of millions of nodes. Training GNN models on such large graphs efficiently remains a big challenge. Despite a number of sampling-based methods have been proposed to enable mini-batch training on large graphs, these methods have not been proved to work on truly industry-scale graphs, which require GPUs or mixed-CPU-GPU training. The state-of-the-art sampling-based methods are usually not optimized for these real-world hardware setups, in which data movement between CPUs and GPUs is a bottleneck. To address this issue, we propose Global Neighborhood Sampling that aims at training GNNs on giant graphs specifically for mixed-CPU-GPU training. The algorithm samples a global cache of nodes periodically for all mini-batches and stores them in GPUs. This global cache allows in-GPU importance sampling of mini-batches, which drastically reduces the number of nodes in a mini-batch, especially in the input layer, to reduce data copy between CPU and GPU and mini-batch computation without compromising the training convergence rate or model accuracy. We provide a highly efficient implementation of this method and show that our implementation outperforms an efficient node-wise neighbor sampling baseline by a factor of 2X-4X on giant graphs. It outperforms an efficient implementation of LADIES with small layers by a factor of 2X-14X while achieving much higher accuracy than LADIES.We also theoretically analyze the proposed algorithm and show that with cached node data of a proper size, it enjoys a comparable convergence rate as the underlying node-wise sampling method.
Data-parallel distributed training of very large models beyond GPU capacity
IBM；IBM Research（IBM研究院）
GPU的内存有限，很难训练宽和/或深的模型，这会导致训练过程耗尽内存。本文展示了一种名为Large Model Support（LMS）的开源工具如何利用CPU和GPU之间的高带宽NVLink连接来完成深度卷积网络的训练。LMS在CPU内存和GPU内存之间执行张量交换，使得在训练步骤中仅将所需的最小数量的张量保留在GPU内存中。它还显示了LMS如何与基于MPI的分布式深度学习模块相结合，以跨多个GPU的数据并行方式训练模型，使得每个GPU都利用CPU内存进行张量交换。讨论了支持高带宽GPU与CPU链接的硬件架构，以及作为PowerAI包提供的相关软件工具集。
ZenFlow: Enabling Stall-Free Offloading for LLM Training
University of Virginia（弗吉尼亚大学）；University of California, Merced（加州大学梅尔塞德斯分校）；Argonne National Laboratory（阿贡国家实验室）；Snowflake AI Research（Snowflake人工智能研究）
Accelerating Reinforcement Learning through GPU Atari Emulation
NVIDIA（英伟达）
我们介绍了CuLE（CUDA学习环境），这是Atari学习环境（ALE）的CUDA端口，用于开发深度强化算法。CuLE克服了现有基于CPU的仿真器的许多限制，并自然地扩展到多个GPU。它利用GPU并行化来同时运行数千个游戏，并直接在GPU上渲染帧，以避免有限的CPU-GPU通信带宽带来的瓶颈。CuLE在单个GPU上每小时生成高达1.55亿帧，这一发现以前只能通过CPU集群实现。除了强调CPU和GPU仿真器在强化学习上下文中的差异之外，我们还展示了如何通过有效地训练训练数据来利用CuLE的高吞吐量，并展示了A2 C +V-trace的加速收敛。CuLE可在https://github.com/NVLabs/cule上获得。
