arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.02852cs.LGcs.AIcs.CLcs.CY
收藏

机器学习模型大小和参数间隙

Machine Learning Model Sizes and the Parameter Gap

Epoch;University of Aberdeen(阿伯丁大学);Centre for the Governance of AI(人工智能治理中心)

Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Anson Ho, Marius Hobbhahn

摘要:

我们使用精心策划的数据集研究了著名的机器学习系统随着时间的推移模型大小的趋势。从1950年到2018年,语言模型的模型规模稳步增长了7个数量级。随后这一趋势加速,从2018年到2022年的短短4年内,模型尺寸又增加了5个数量级。视觉模型以更稳定的速度增长,在1950年至2022年之间总共增长了7个数量级。我们还发现,自2020年以来,20 B参数以下的语言模型很多,70 B参数以上的模型很多,但20- 70 B参数范围内的模型很少。我们把这种稀缺性称为参数缺口。我们提供了一些关于参数差距的程式化事实,并提出了一些假设来解释它。我们喜欢的解释是:(a)将模型大小增加到超过20 B参数需要采用不同的并行技术,这使得中型模型的成本效益较低,(b)GPT-3比以前的语言模型大一个数量级,研究人员后来主要用更大的模型进行实验,以超越它。虽然这些动态可能存在,我们相信它们在产生差距方面发挥了一定作用,但我们不太相信没有其他的,更重要的动力

英文摘要:

We study trends in model size of notable machine learning systems over time using a curated dataset. From 1950 to 2018, model size in language models increased steadily by seven orders of magnitude. The trend then accelerated, with model size increasing by another five orders of magnitude in just 4 years from 2018 to 2022. Vision models grew at a more constant pace, totaling 7 orders of magnitude of growth between 1950 and 2022. We also identify that, since 2020, there have been many language models below 20B parameters, many models above 70B parameters, but a scarcity of models in the 20-70B parameter range. We refer to that scarcity as the parameter gap. We provide some stylized facts about the parameter gap and propose a few hypotheses to explain it. The explanations we favor are: (a) increasing model size beyond 20B parameters requires adopting different parallelism techniques, which makes mid-sized models less cost-effective, (b) GPT-3 was one order of magnitude larger than previous language models, and researchers afterwards primarily experimented with bigger models to outperform it. While these dynamics likely exist, and we believe they play some role in generating the gap, we don't have high confidence that there are no other, more important dynamics at play.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑