arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.11853cs.LGcs.AI
收藏

深度学习工作负载下的GPU故障预测

Prediction of GPU Failures Under Deep Learning Workloads

The Pennsylvania State University(宾夕法尼亚州立大学);ByteDance(字节跳动);Northeastern University(东北大学)

Heting Liu, Zhichao Li, Cheng Tan, Rongqiu Yang, Guohong Cao, Zherui Liu, Chuanxiong Guo

摘要:

图形处理单元(GPU)是处理深度学习(DL)任务的事实标准。与此同时,不可避免的GPU故障会在DL任务中造成严重后果:它们会中断分布式训练,崩溃推理服务,并导致违反服务级别协议。为了减轻GPU故障引起的问题,我们建议使用ML模型来预测故障。本文是第一篇研究大规模生产深度学习工作负载下GPU故障预测模型的论文。首先,我们评估了经典的预测模型,并观察到这些模型的预测既不准确又不稳定。为了提高预测的精度和稳定性,我们提出了几种技术,包括并行和级联模型集成机制和滑动训练方法。我们评估了我们的各种技术在四个月的生产数据集,包括3.5亿个条目的性能。结果表明,该方法将预测精度从46.3%提高到84.0%。

英文摘要:

Graphics processing units (GPUs) are the de facto standard for processing deep learning (DL) tasks. Meanwhile, GPU failures, which are inevitable, cause severe consequences in DL tasks: they disrupt distributed trainings, crash inference services, and result in service level agreement violations. To mitigate the problem caused by GPU failures, we propose to predict failures by using ML models. This paper is the first to study prediction models of GPU failures under large-scale production deep learning workloads. As a starting point, we evaluate classic prediction models and observe that predictions of these models are both inaccurate and unstable. To improve the precision and stability of predictions, we propose several techniques, including parallel and cascade model-ensemble mechanisms and a sliding training method. We evaluate the performances of our various techniques on a four-month production dataset including 350 million entries. The results show that our proposed techniques improve the prediction precision from 46.3\% to 84.0\%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑