声学模型的标度律
Scaling Laws for Acoustic Models
摘要：
机器学习最近有一种趋势，即通过将模型增长到以前认为不合理的大小来提高模型质量。最近的工作表明，具有交叉熵目标函数的自回归生成模型表现出平滑的幂律关系或缩放律，可以从模型大小，训练集大小和可用的计算预算预测模型质量。这些比例律允许人们在给定对可用训练数据、模型参数计数或训练计算预算的约束的情况下选择接近最优的超参数。在本文中，我们证明了用自动预测编码损失训练的声学模型的行为就好像它们受到类似的缩放律。我们扩展了以前的工作，共同预测由于模型大小，训练集大小和任务固有的“不可减少的损失”造成的损失。我们发现，比例律准确地匹配模型的性能超过两个数量级的模型大小和训练集大小，并预测模型性能的限制。
英文摘要：
There is a recent trend in machine learning to increase model quality by growing models to sizes previously thought to be unreasonable. Recent work has shown that autoregressive generative models with cross-entropy objective functions exhibit smooth power-law relationships, or scaling laws, that predict model quality from model size, training set size, and the available compute budget. These scaling laws allow one to choose nearly optimal hyper-parameters given constraints on available training data, model parameter count, or training computation budget. In this paper, we demonstrate that acoustic models trained with an auto-predictive coding loss behave as if they are subject to similar scaling laws. We extend previous work to jointly predict loss due to model size, to training set size, and to the inherent "irreducible loss" of the task. We find that the scaling laws accurately match model performance over two orders of magnitude in both model size and training set size, and make predictions about the limits of model performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
