arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.07821cs.LGcs.PF
收藏

在GPU上构建深度学习推荐模型训练的性能模型

Building a Performance Model for Deep Learning Recommendation Model Training on GPUs

University of California, Davis(加州大学戴维斯分校);Meta Platforms, Inc.(Meta平台公司)

Zhongyi Lin, Louis Feng, Ehsan K. Ardestani, Jaewon Lee, John Lundell, Changkyu Kim, Arun Kejariwal, John D. Owens

摘要:

我们为深度学习推荐模型(DLRM)的GPU训练设计了一个性能模型,与其他优化良好的CV和NLP模型相比,DLRM的GPU利用率较低。我们表明,无论是设备的活动时间(内核运行时的总和),但也设备空闲时间的重要组成部分,整体设备的时间。因此,我们通过以下方式分别解决它们:(1)针对主导设备活动时间的运营商灵活地采用基于机器学习和基于ML的内核性能模型,以及(2)将运营商开销分为五种类型,以定量地确定它们对设备活动时间的贡献。结合这两个部分,我们提出了一个基于关键路径的算法来预测每批训练时间的DLRM通过遍历其执行图。我们在所有内核性能建模中实现了不到10%的几何平均误差(GMAE),GPU活动时间和整体E2E每批训练时间预测的几何误差分别为4.61%和7.96%,其中每个工作负载的开销。E2E预测误差略微增加2.19%,跨工作负载共享开销表明在大规模预测中使用共享开销的可行性。我们表明,我们的通用性能模型不仅在DLRM上实现了较低的预测误差,DLRM具有高度定制的配置,并受多个因素的影响,而且在大多数以前的方法所针对的其他计算约束ML模型上也具有相当的准确性。使用此性能模型和图级数据和任务依赖性分析,我们表明我们的系统可以提供更一般的模型系统协同设计比以前的方法。

英文摘要:

We devise a performance model for GPU training of Deep Learning Recommendation Models (DLRM), whose GPU utilization is low compared to other well-optimized CV and NLP models. We show that both the device active time (the sum of kernel runtimes) but also the device idle time are important components of the overall device time. We therefore tackle them separately by (1) flexibly adopting heuristic-based and ML-based kernel performance models for operators that dominate the device active time, and (2) categorizing operator overheads into five types to determine quantitatively their contribution to the device active time. Combining these two parts, we propose a critical-path-based algorithm to predict the per-batch training time of DLRM by traversing its execution graph. We achieve less than 10% geometric mean average error (GMAE) in all kernel performance modeling, and 4.61% and 7.96% geomean errors for GPU active time and overall E2E per-batch training time prediction with overheads from individual workloads, respectively. A slight increase of 2.19% incurred in E2E prediction error with shared overheads across workloads suggests the feasibility of using shared overheads in large-scale prediction. We show that our general performance model not only achieves low prediction error on DLRM, which has highly customized configurations and is dominated by multiple factors but also yields comparable accuracy on other compute-bound ML models targeted by most previous methods. Using this performance model and graph-level data and task dependency analysis, we show our system can provide more general model-system co-design than previous methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑