Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay ($λ=1.0$) reduces this coefficient by approximately 70\%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.
Effective Frontiers: A Unification of Neural Scaling Laws
Department of Mathematics and Statistics, Xi'an Jiaotong University(西安交通大学数学与统计学系);Department of Artificial Intelligence, Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);University of Chinese Academy of Sciences(中国科学院大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Projected Neural Differential Equations for Learning Constrained Dynamics
Technical University of Munich(慕尼黑工业大学);Potsdam Institute for Climate Impact Research(波茨坦气候影响研究所);California Institute of Technology(加州理工学院);Helmholtz Munich(亥姆霍兹慕尼黑中心);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);University of Exeter(埃克塞特大学)
Research School of Astronomy & Astrophysics, The Australian National University(澳大利亚国立大学天文与天体物理研究所);Astronomical Institute, University of Wrocław(沃克劳大学天文研究所);Department of Astronomy, The Ohio State University(俄亥俄州立大学天文学系);Center for Cosmology and AstroParticle Physics (CCAPP), The Ohio State University(宇宙与天体粒子物理中心(CCAPP),俄亥俄州立大学)
本文研究 Transformer 恒星光谱仿真器的神经缩放定律,结合 TransformerPayne 与 Maximum Update Parametrization 给出均衡扩展训练指南,为光谱基础模型奠定基础。
Determinant Estimation under Memory Constraints and Neural Scaling Laws
Department of Statistics, University of California, Berkeley CA, USA(加州大学伯克利分校统计学系);International Computer Science Institute, Berkeley CA, USA(国际计算机科学研究所);Electronic Engineering, University of Melbourne, Australia(墨尔本大学电子工程系);School of Mathematics(数学学院);Statistics, University of Melbourne, Australia(统计学,墨尔本大学);Physics, University of Queensland, Australia(物理,昆士兰大学);Lawrence Berkeley National Laboratory, Berkeley CA, USA(劳伦斯伯克利国家实验室)