arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.08679cs.IRcs.LG
收藏

Check-N-Run:一个用于训练深度学习推荐模型的检查点系统

Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models

Facebook, Inc(Facebook公司);USC(南加州大学)

Assaf Eisenman, Kiran Kumar Matam, Steven Ingram, Dheevatsa Mudigere, Raghuraman Krishnamoorthi, Krishnakumar Nair, Misha Smelyanskiy, Murali Annavaram

摘要:

检查点在训练长时间运行的机器学习(ML)模型中起着重要作用。检查点获取ML模型的快照并将其存储在非易失性存储器中,以便它们可以用于从故障中恢复,以确保快速的训练进度。此外,它们还用于在线训练,以通过持续学习提高推理预测的准确性。由于模型大小较大且不断增加,检查点频率通常会受到存储写入带宽和容量的影响。当在远程存储上维护检查点时,就像许多工业设置一样,它们也会受到网络带宽的检查。我们提出了Check-N-Run,这是一个可扩展的检查点系统,用于在Facebook上训练大型ML模型。虽然Check-N-Run适用于长时间运行的机器学习作业,但我们专注于检查点推荐模型,这是目前最大的机器学习模型,模型大小为TB。Check-N-Run使用两种主要技术来解决大小和带宽挑战。首先,它应用增量检查点,跟踪和检查模型的修改部分。增量检查点在推荐模型的上下文中特别有价值,在推荐模型中,每次迭代只更新一小部分模型(存储为嵌入表)。其次,Check-N-Run利用量化技术显著减少检查点大小,而不会降低训练精度。这些技术允许Check-N-Run在Facebook的真实世界模型上将所需的写入带宽减少6- 17倍,将所需的容量减少2.5- 8倍,从而显著提高检查点功能,同时降低总拥有成本。

英文摘要:

Checkpoints play an important role in training long running machine learning (ML) models. Checkpoints take a snapshot of an ML model and store it in a non-volatile memory so that they can be used to recover from failures to ensure rapid training progress. In addition, they are used for online training to improve inference prediction accuracy with continuous learning. Given the large and ever increasing model sizes, checkpoint frequency is often bottlenecked by the storage write bandwidth and capacity. When checkpoints are maintained on remote storage, as is the case with many industrial settings, they are also bottlenecked by network bandwidth. We present Check-N-Run, a scalable checkpointing system for training large ML models at Facebook. While Check-N-Run is applicable to long running ML jobs, we focus on checkpointing recommendation models which are currently the largest ML models with Terabytes of model size. Check-N-Run uses two primary techniques to address the size and bandwidth challenges. First, it applies incremental checkpointing, which tracks and checkpoints the modified part of the model. Incremental checkpointing is particularly valuable in the context of recommendation models where only a fraction of the model (stored as embedding tables) is updated on each iteration. Second, Check-N-Run leverages quantization techniques to significantly reduce the checkpoint size, without degrading training accuracy. These techniques allow Check-N-Run to reduce the required write bandwidth by 6-17x and the required capacity by 2.5-8x on real-world models at Facebook, and thereby significantly improve checkpoint capabilities while reducing the total cost of ownership.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑