arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.01640cs.LGcs.CL
收藏

数据受限训练的规范性扩展定律

Prescriptive Scaling Laws for Data Constrained Training

Department of Computer Science(计算机科学系)

Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, Kilian Q. Weinberger

AI总结:

本文提出新的扩展定律,通过引入过拟合惩罚,改进数据受限训练中的计算分配策略,证明过度重复无效,建议增加模型容量,且通过权重衰减系数比较验证其有效性。

摘要:

训练计算量正日益超过高质量数据的可用性。这将核心挑战从最优计算分配转向从有限数据中提取最大价值。广泛采用的Chinchilla扩展定律假设每个训练token都是唯一的,这限制了其在数据受限环境中的预训练决策指导能力。我们通过简单的加性过拟合惩罚模型,发现其能准确描述模型行为。我们的扩展定律提供了新的计算最优分配建议。在某个点之后,进一步重复是无益的,计算应更多用于模型容量。我们证明遵循该定律推荐的配置在数据受限环境中能提升性能。最后,由于我们的单参数形式将过拟合隔离在单一系数中,它使不同训练配置之间的直接比较成为可能。作为案例研究,我们展示强权重衰减(λ=1.0)可将此系数降低约70%,为最近发现的数据受限环境中最优权重衰减比标准实践大一个数量级提供了扩展定律解释。

英文摘要:

Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay ($λ=1.0$) reduces this coefficient by approximately 70\%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑