微观尺度下的缩放定律：从小规模实验预测Transformer性能
Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments
摘要：
神经缩放定律以幂律的形式定义了模型参数计数与训练后性能之间的可预测关系。然而，迄今为止，大多数研究还没有明确调查是否缩放定律可以用来加速模型的发展。在这项工作中，我们在广泛的语言理解任务中进行了这样的实证调查，从具有少至10K参数的模型开始，并在9个语言理解任务中评估下游性能。我们发现，缩放定律出现在一些NLP任务中的微调时间，并且它们也可以用于训练大型模型时的调试收敛。此外，对于存在缩放定律的任务，它们可以用于预测更大模型的性能，从而实现有效的模型选择。然而，揭示缩放定律需要仔细的超参数调整和多次运行的目的不确定性估计，这会产生额外的开销，部分抵消了计算的好处。
英文摘要：
Neural scaling laws define a predictable relationship between a model's parameter count and its performance after training in the form of a power law. However, most research to date has not explicitly investigated whether scaling laws can be used to accelerate model development. In this work, we perform such an empirical investigation across a wide range of language understanding tasks, starting from models with as few as 10K parameters, and evaluate downstream performance across 9 language understanding tasks. We find that scaling laws emerge at finetuning time in some NLP tasks, and that they can also be exploited for debugging convergence when training large models. Moreover, for tasks where scaling laws exist, they can be used to predict the performance of larger models, which enables effective model selection. However, revealing scaling laws requires careful hyperparameter tuning and multiple runs for the purpose of uncertainty estimation, which incurs additional overhead, partially offsetting the computational benefits.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
