arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.04747cs.DCcs.NE
收藏

在 GPU 上实现数据高效深度学习训练的代码生成与运行时技术

Code generation and runtime techniques for enabling data-efficient deep learning training on GPUs

Kun Wu

备注：
Ph.D. Thesis, University of Illinois Urbana-Champaign, 2024
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文针对 GNN 与 LLM 训练中的数据移动、PCIe 传输和 GPU 显存瓶颈，提出 PyTorch-Direct、Hector IR 代码生成器及 SSDTrain 卸载框架。

摘要：

随着深度学习模型规模扩大，其训练成本显著激增。由于硬件进步以及当前软件栈的局限，数据效率的需求已经上升。数据效率指的是有效隐藏数据访问延迟并避免不必要的数据移动。主要挑战来自 GPU 内存带宽与计算吞吐量之间日益扩大的差距、迫在眉睫的 GPU 内存容量限制，以及 PyTorch 软件栈中的低效问题，包括缺少面向特定设备的 PCIe 传输优化和高级领域专用抽象。为有效缓解深度学习训练中的这些数据低效问题，本论文分析了代表性深度训练任务中的数据低效，具体包括图神经网络（GNN）和大语言模型（LLM）。随后，论文提出新颖的运行时与代码生成技术以缓解这些挑战，并在 PyTorch 栈内无缝实现这些优化，同时保持强大的可编程性和互操作性。首先，设计 PyTorch-Direct，以在用于 GNN 训练的 PyTorch 中纳入以 GPU 为中心的 PCIe 数据传输范式。接着，提出 Hector 中间表示（IR）及其代码生成器，以引入领域专用的高级抽象，并系统性解决关系型 GNN 中内存密集型的性能挑战。最后，在 LLM 训练中，吞吐量日益受到 GPU 内存容量的约束。为缓解这一问题，设计并实现 SSDTrain 卸载框架。这些贡献共同表明，代码生成与运行时技术能够系统性缓解深度学习训练中的数据管理瓶颈；这些瓶颈源于工作负载的数据密集型本质，以及深度学习训练软件栈固有的过度简化。

英文摘要：

As deep learning models scale, their training cost has surged significantly. Due to both hardware advancements and limitations in current software stacks, the need for data efficiency has risen. Data efficiency refers to the effective hiding of data access latency and the avoidance of unnecessary data movements. Major challenges arise from the growing disparity between GPU memory bandwidth and computational throughput, imminent GPU memory capacity limitations, and inefficiencies in the PyTorch software stack, including a lack of device-specific PCIe transfer optimizations and high-level domain-specific abstractions. To effectively mitigate these data inefficiencies for deep learning training, this dissertation analyzes data inefficiency in representative deep training tasks, specifically in graph neural networks (GNNs) and large language models (LLMs). It then proposes novel runtime and code generation techniques to mitigate these challenges and implements these optimizations seamlessly within the PyTorch stack while maintaining strong programmability and interoperability. First, PyTorch-Direct is devised to incorporate the GPU-centric PCIe data transfer paradigm in PyTorch for GNN training. Next, Hector intermediate representation (IR) and its code generator are proposed to introduce domain-specific high-level abstraction and systematically address memory-intensive performance challenges for relational GNNs. Finally, in LLM training, the throughput has been increasingly constrained by GPU memory capacity. To mitigate this, the SSDTrain offloading framework is designed and implemented. Together, these contributions show that code generation and runtime techniques can systematically mitigate the data management bottlenecks in deep learning training, which stem from the data-intensive nature of workloads and the oversimplification inherent in the deep learning training software stack.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑