arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.11928stat.MLcs.LG
收藏

可训练参数的数量真的很重要吗?

Is the Number of Trainable Parameters All That Actually Matters?

LightOn;Sorbonne Université(索邦大学);LPENS, École Normale Supérieure(高等师范学院自然科学系实验室)

Amélie Chatelain, Amine Djeghri, Daniel Hesslow, Julien Launay, Iacopo Poli

摘要:

最近的工作已经确定了语言模型的简单经验缩放定律,将计算预算,数据集大小,模型大小和自回归建模损失联系起来。这些简单的幂律在模型尺度上跨越数量级的有效性提供了令人信服的证据,证明更大的模型也是更有能力的模型。然而,在硬件和基础设施的限制下扩大模型并不是一件容易的事情,并且很快成为一个困难而昂贵的工程问题。我们研究了尝试性地欺骗标度律的方法,并训练更大的模型以获得更便宜的结果。我们使用有效的近似来模拟有效参数的增加:通过使用冻结的随机参数来掺杂模型,或者通过使用快速结构化变换来代替密集的线性层。我们发现,测试损失和计算之间的标度关系只取决于实际的可训练参数的数量,标度律不能欺骗虚假参数。

英文摘要:

Recent work has identified simple empirical scaling laws for language models, linking compute budget, dataset size, model size, and autoregressive modeling loss. The validity of these simple power laws across orders of magnitude in model scale provides compelling evidence that larger models are also more capable models. However, scaling up models under the constraints of hardware and infrastructure is no easy feat, and rapidly becomes a hard and expensive engineering problem. We investigate ways to tentatively cheat scaling laws, and train larger models for cheaper. We emulate an increase in effective parameters, using efficient approximations: either by doping the models with frozen random parameters, or by using fast structured transforms in place of dense linear layers. We find that the scaling relationship between test loss and compute depends only on the actual number of trainable parameters; scaling laws cannot be deceived by spurious parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑