arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.08702cs.CL
收藏

代码的缩放规律:一个更数据渴求的阶段

Scaling Laws for Code: A More Data-Hungry Regime

Harbin Institute of Technology(哈尔滨工业大学);Chinese Academy of Sciences(中国科学院);Fudan University(复旦大学);Beijing University of Posts and Telecommunications(北京邮电大学)

Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu, Rongyi Zhang, Houyi Li, Siming Huang, YuanTao Fan, Wanxiang Che

AI总结:

本文研究了代码的缩放规律,通过大规模实验发现Farseer定律在准确性上更优,代码模型在模型大小上表现良好,但需要更高的数据与参数比,且在代码-自然语言混合数据中,自然语言在资源受限场景下有益,但在更高计算预算下成为负担。

摘要:

代码大型语言模型(LLMs)正在革新软件工程。然而,指导高效训练的缩放定律主要是在自然语言(NL)上分析的。鉴于代码和自然语言之间的根本差异,如严格的语法,这些定律是否直接适用于代码尚不清楚。为填补这一空白,我们进行了首次大规模的代码缩放定律实证研究,包括117次实验运行,模型大小从0.2B到3.8B,训练token从2B到128B。我们拟合了Chinchilla定律和Farsser定律。首先,结果表明,更具表现力的Farsser定律在准确性上更优。其次,分析显示代码LLMs在模型大小上有效扩展。关键的是,代码代表了一个更数据渴求的阶段,需要比自然语言显著更高的数据与参数比。最后,对代码-自然语言混合数据的两个额外实验显示,自然语言在资源受限的场景下有益,但在更高计算预算下成为负担。

英文摘要:

Code Large Language Models (LLMs) are revolutionizing software engineering. However, scaling laws that guide the efficient training are predominantly analyzed on Natural Language (NL). Given the fundamental differences like strict syntax between code and NL, it is unclear whether these laws are directly applicable to code. To address this gap, we conduct the first large-scale empirical study of scaling laws for code, comprising 117 experimental runs with model sizes from 0.2B to 3.8B and training tokens from 2B to 128B. We fit the Chinchilla law and the Farsser law. First, the results show that the more expressive Farseer law offers greater accuracy. Second, the analysis reveals that Code LLMs scale effectively with model size. Crucially, code represents a more data-hungry regime, requiring a substantially higher data-to-parameter ratio than NL. Finally, two additional sets of experiments on code-NL mixtures show that NL benefits resource-constrained scenarios, but becomes a detriment at higher compute budgets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑