导航缩放定律:自适应模型训练中的计算最优性
Navigating Scaling Laws: Compute Optimality in Adaptive Model Training
AI总结:
本文提出自适应模型训练方法,通过动态调整模型形状在缩放定律间最优穿行,超越静态模型,显著降低达到目标性能所需的计算量,并跨模态泛化。
摘要:
近年来,深度学习的最先进技术一直被在海量数据上预训练的超大模型所主导。其范式非常简单:投入更多(最优的)计算资源会带来更好的性能,甚至这种提升是可预测的;神经缩放定律已被推导出来,能够准确预测网络在期望计算水平下的性能。这引出了“计算最优”模型的概念,即一种在训练过程中分配给定计算水平以最大化性能的模型。在这项工作中,我们通过允许“自适应”模型(即能够在训练过程中改变其形状的模型)来扩展最优性的概念。通过这样做,我们可以设计出自适应模型,使其在底层缩放定律之间最优地穿行,并超越其“静态”对应模型,从而在达到给定目标性能时显著减少所需计算量。我们展示了我们的方法跨模态和不同形状参数具有泛化能力。
英文摘要:
In recent years, the state-of-the-art in deep learning has been dominated by very large models that have been pre-trained on vast amounts of data. The paradigm is very simple: investing more computational resources (optimally) leads to better performance, and even predictably so; neural scaling laws have been derived that accurately forecast the performance of a network for a desired level of compute. This leads to the notion of a `compute-optimal' model, i.e. a model that allocates a given level of compute during training optimally to maximize performance. In this work, we extend the concept of optimality by allowing for an `adaptive' model, i.e. a model that can change its shape during training. By doing so, we can design adaptive models that optimally traverse between the underlying scaling laws and outpace their `static' counterparts, leading to a significant reduction in the required compute to reach a given target performance. We show that our approach generalizes across modalities and different shape parameters.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。