arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2208.08489cs.IRcs.LG
收藏

理解推荐模型的缩放定律

Understanding Scaling Laws for Recommendation Models

Meta

Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, Adnan Aziz

摘要:

规模一直是提高机器学习性能的主要驱动力,理解缩放定律对于可持续模型质量性能增长的战略规划、长期资源规划和开发有效的系统基础设施以支持大规模模型至关重要。本文研究了DLRM风格推荐模型的经验缩放定律,特别是点击率(CTR)。我们观察到模型质量与幂律以及模型大小、数据大小和用于训练的计算量的常数有关。我们沿着三个不同的资源维度,即数据,参数和计算,通过比较不同的缩放方案沿这些轴的缩放效率。我们表明,参数缩放是蒸汽的模型架构正在研究中,直到一个更高性能的模型架构出现,数据缩放是前进的道路。本研究所解决的关键研究问题包括:一个推荐模型的规模可持续的预测的缩放定律?或者我们离缩放定律的预测还很远?缩放的极限是什么?缩放定律对长期硬件/系统开发有何影响?

英文摘要:

Scale has been a major driving force in improving machine learning performance, and understanding scaling laws is essential for strategic planning for a sustainable model quality performance growth, long-term resource planning and developing efficient system infrastructures to support large-scale models. In this paper, we study empirical scaling laws for DLRM style recommendation models, in particular Click-Through Rate (CTR). We observe that model quality scales with power law plus constant in model size, data size and amount of compute used for training. We characterize scaling efficiency along three different resource dimensions, namely data, parameters and compute by comparing the different scaling schemes along these axes. We show that parameter scaling is out of steam for the model architecture under study, and until a higher-performing model architecture emerges, data scaling is the path forward. The key research questions addressed by this study include: Does a recommendation model scale sustainably as predicted by the scaling laws? Or are we far off from the scaling law predictions? What are the limits of scaling? What are the implications of the scaling laws on long-term hardware/system development?

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑