arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.01515cs.IRcs.LGcs.PF
收藏

MTrainS: 利用异构内存提升DLRM训练效率

MTrainS: Improving DLRM training efficiency using heterogeneous memories

Meta Platforms, Inc.(元平台公司);University of Michigan(密歇根大学)

Hiwot Tadese Kassa, Paul Johnson, Jason Akers, Mrinmoy Ghosh, Andrew Tulloch, Dheevatsa Mudigere, Jongsoo Park, Xing Liu, Ronald Dreslinski, Ehsan K. Ardestani

AI总结:

针对DLRM训练内存需求大、资源消耗高的问题,提出MTrainS框架分层利用异构内存,通过利用嵌入表的时间局部性优化内存层级,减少4-8倍训练节点数,在保障性能的同时降低功耗与成本。

摘要:

推荐模型规模极大,训练期间需要数太字节(TB)的内存。为追求更优效果,模型规模与复杂度随时间不断增长,这就需要更多训练数据以避免过拟合。这种模型增长给数据中心带来了大量资源需求。因此,提升训练效率对于管控数据中心的电力需求正变得愈发重要。在深度学习推荐模型(Deep Learning Recommendation Model,DLRM)中,通过嵌入表捕获类别输入的稀疏特征是模型规模的主要构成部分,且需要高内存带宽。本文研究了实际部署模型中嵌入表的带宽需求与局部性。我们观察到,不同表的带宽需求并不均匀,且嵌入表表现出较高的时间局部性。随后我们设计了MTrainS框架,该框架分层利用异构内存(包括字节可寻址与块可寻址的存储级内存(Storage Class Memory))来支撑DLRM训练。MTrainS提升了单节点内存容量,在受内存容量限制的应用场景中,降低了横向扩展至多主机的需求,从而提升训练效率。通过优化平台内存层级,我们将训练所需的节点数减少了4-8倍,在满足目标训练性能的同时,节省了训练的电力与成本。

英文摘要:

Recommendation models are very large, requiring terabytes (TB) of memory during training. In pursuit of better quality, the model size and complexity grow over time, which requires additional training data to avoid overfitting. This model growth demands a large number of resources in data centers. Hence, training efficiency is becoming considerably more important to keep the data center power demand manageable. In Deep Learning Recommendation Models (DLRM), sparse features capturing categorical inputs through embedding tables are the major contributors to model size and require high memory bandwidth. In this paper, we study the bandwidth requirement and locality of embedding tables in real-world deployed models. We observe that the bandwidth requirement is not uniform across different tables and that embedding tables show high temporal locality. We then design MTrainS, which leverages heterogeneous memory, including byte and block addressable Storage Class Memory for DLRM hierarchically. MTrainS allows for higher memory capacity per node and increases training efficiency by lowering the need to scale out to multiple hosts in memory capacity bound use cases. By optimizing the platform memory hierarchy, we reduce the number of nodes for training by 4-8X, saving power and cost of training while meeting our target training performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑