arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.05924cs.LGcs.AIcs.CY
收藏

机器学习三个时代的计算趋势

Compute Trends Across Three Eras of Machine Learning

University of Aberdeen(阿伯丁大学);Centre for the Governance of AI(人工智能治理中心);University of St Andrews(圣安德鲁斯大学);Massachusetts Institute of Technology(麻省理工学院);University of Tübingen(蒂宾根大学);Complutense University of Madrid(马德里康普顿斯大学)

Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos

AI总结:

本文研究机器学习训练计算的增长趋势,将其历史划分为深度学习前时代、深度学习时代和大规模时代,揭示了先进ML系统对计算需求的快速增长。

摘要:

计算、数据和算法进步是推动现代机器学习(ML)进步的三个基本因素。本文研究了最容易量化的因素——计算。我们表明,在2010年之前,训练计算的增长与摩尔定律一致,大约每20个月翻一番。自2010年代初深度学习出现以来,训练计算的扩展加速,大约每6个月翻一番。2015年底,随着企业开发出训练计算需求大10到100倍的大规模ML模型,新趋势出现。基于这些观察,我们将ML中的计算历史分为三个时代:深度学习前时代、深度学习时代和大规模时代。总体而言,我们的工作突出了训练先进ML系统对计算需求的快速增长。

英文摘要:

Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore's law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute. Based on these observations we split the history of compute in ML into three eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑