arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.01468cs.DCcs.AIcs.DB
收藏

DLRover-RM:云中深度推荐模型训练的资源优化

DLRover-RM: Resource Optimization for Deep Recommendation Models Training in the Cloud

Sichuan University(四川大学);The University of Texas at Arlington(德克萨斯大学阿灵顿分校);Computer Network Information Center, Chinese Academy of Science(中国科学院计算机网络信息中心)

Qinlong Wang, Tingfeng Lan, Yinghao Tang, Ziling Huang, Yiheng Du, Haitao Zhang, Jian Sha, Hui Lu, Yuanchun Zhou, Ke Zhang, Mingjie Tang

AI总结:

本文针对云环境中DLRM训练资源利用率低和不稳定的问题,提出弹性训练框架DLRover-RM,通过资源-性能模型与三阶段启发式策略自动分配和动态调整资源,显著缩短作业完成时间并提升资源利用率。

摘要:

深度学习推荐模型(DLRM)依赖大型嵌入表来管理类别型稀疏特征。扩展此类嵌入表可以显著提升模型性能,但代价是增加GPU/CPU/内存使用。与此同时,科技公司已构建了广泛的云服务以加速大规模训练DLRM模型。在本文中,我们对AntGroup的DLRM训练平台进行了深入调查,并揭示两个关键挑战:由于用户配置不理想导致的资源利用率低下,以及由于不稳定的云环境而倾向于出现异常。为克服这些问题,我们提出DLRover-RM,这是一个面向DLRM的弹性训练框架,旨在提高资源利用率并应对云环境的不稳定性。DLRover-RM通过考虑DLRM的独特特性构建资源-性能模型,并采用三阶段启发式策略自动分配和动态调整DLRM训练作业的资源,以实现更高的资源利用率。此外,DLRover-RM开发了多种机制来确保DLRM训练作业的高效可靠执行。我们的大量评估表明,与最先进的资源调度框架相比,DLRover-RM将作业完成时间缩短31%,将作业完成率提高6%,将CPU使用率提升15%,并将内存利用率提高20%。DLRover-RM已在AntGroup广泛部署,每天处理数千个DLRM训练作业。DLRover-RM已开源,并被10多家公司采用。

英文摘要:

Deep learning recommendation models (DLRM) rely on large embedding tables to manage categorical sparse features. Expanding such embedding tables can significantly enhance model performance, but at the cost of increased GPU/CPU/memory usage. Meanwhile, tech companies have built extensive cloud-based services to accelerate training DLRM models at scale. In this paper, we conduct a deep investigation of the DLRM training platforms at AntGroup and reveal two critical challenges: low resource utilization due to suboptimal configurations by users and the tendency to encounter abnormalities due to an unstable cloud environment. To overcome them, we introduce DLRover-RM, an elastic training framework for DLRMs designed to increase resource utilization and handle the instability of a cloud environment. DLRover-RM develops a resource-performance model by considering the unique characteristics of DLRMs and a three-stage heuristic strategy to automatically allocate and dynamically adjust resources for DLRM training jobs for higher resource utilization. Further, DLRover-RM develops multiple mechanisms to ensure efficient and reliable execution of DLRM training jobs. Our extensive evaluation shows that DLRover-RM reduces job completion times by 31%, increases the job completion rate by 6%, enhances CPU usage by 15%, and improves memory utilization by 20%, compared to state-of-the-art resource scheduling frameworks. DLRover-RM has been widely deployed at AntGroup and processes thousands of DLRM training jobs on a daily basis. DLRover-RM is open-sourced and has been adopted by 10+ companies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑