神经缩放定律的动力学模型
A Dynamical Model of Neural Scaling Laws
AI总结：
该研究提出一种基于梯度下降训练的随机特征动力学模型，解释神经缩放定律中不同幂律指数差异、非对称计算最优缩放规则及训练收敛速率的宽度依赖性，还阐明数据复用导致训练与测试损失差距随时间累积的机制。
摘要：
在多种任务中，神经网络的性能会随着训练时长、数据集规模和模型规模跨越多个数量级的提升而可预测地改善，这一现象被称为神经缩放定律。其中计算最优缩放定律具有核心重要性，它描述了在最优选择模型规模时，性能作为计算量单位的函数的变化规律。我们分析了一个用梯度下降训练的随机特征模型，将其作为可求解的网络训练与泛化模型，该模型复现了许多关于神经缩放定律的观测结果。首先，我们的模型对性能随训练时长的缩放与随模型规模的缩放具有不同幂律指数的原因给出了预测。由此，该理论推导出一种非对称的计算最优缩放规则：训练步数的增长速度快于模型参数的增长速度，这与近期的实验观测结果一致。其次，已有研究观测到，在训练早期，网络以$1/\textit{width}$的速率收敛到其无限宽度动力学，但在训练后期则呈现出$\textit{width}^{-c}$的收敛速率，其中$c$取决于架构结构与任务特性。我们证明了我们的模型能够展现出这种行为。最后，我们的理论揭示了训练损失与测试损失之间的差距如何因数据的重复使用而随时间逐渐增大。
英文摘要：
On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude. This phenomenon is known as a neural scaling law. Of fundamental importance is the compute-optimal scaling law, which reports the performance as a function of units of compute when choosing model sizes optimally. We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. This reproduces many observations about neural scaling laws. First, our model makes a prediction about why the scaling of performance with training time and with model size have different power law exponents. Consequently, the theory predicts an asymmetric compute-optimal scaling rule where the number of training steps are increased faster than model parameters, consistent with recent empirical observations. Second, it has been observed that early in training, networks converge to their infinite-width dynamics at a rate $1/\textit{width}$ but at late time exhibit a rate $\textit{width}^{-c}$, where $c$ depends on the structure of the architecture and task. We show that our model exhibits this behavior. Lastly, our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
