在合成编辑序列上训练语言模型可提升代码合成性能
Training Language Models on Synthetic Edit Sequences Improves Code Synthesis
AI总结:
针对代码合成中顺序编辑数据稀缺问题,提出合成数据生成算法LintSeq,通过代码检查工具重构程序为合成编辑序列,微调后的模型在多项基准上匹配或优于基线,且性能优于类似规模现有模型。
摘要:
软件工程师主要通过编辑现有程序编写代码,而语言模型(LMs)则以自回归方式一次性合成程序。对此的一种解释是顺序编辑数据稀缺。虽然代码合成的高质量指令数据稀缺,但用于合成的编辑数据更为稀缺。为填补这一空白,我们开发了一种名为LintSeq的合成数据生成算法。该算法通过使用代码检查工具(linter)对源代码中相互依赖的行进行程序化采样,将程序重构为合成编辑序列。LintSeq采样的合成编辑反映了其编程语言的语法和语义。为测试该算法,我们用它将指令+程序对数据集重构为指令+程序差异序列元组,然后在重构和原始版本的数据集上微调一系列参数规模从26亿到140亿的较小语言模型。我们在HumanEval、MBPP(+)、CodeContests、DS-1000和BigCodeBench上进行综合评估,比较基于编辑序列的代码语言模型与基线模型。结果显示,经迭代合成代码微调的模型在pass@1上匹配或优于基线模型,且在更高的pass@k上随测试时总浮点运算次数(FLOPs)展现出更好的缩放性能。最后,我们还预训练了用于代码理解的小型语言模型,结果表明,将这些模型微调为逐编辑合成代码后,在HumanEval和MBPP(+)上的性能优于CodeT5+、AlphaCode和Codex等类似规模的现有代码语言模型。
英文摘要:
Software engineers mainly write code by editing existing programs. In contrast, language models (LMs) autoregressively synthesize programs in a single pass. One explanation for this is the scarcity of sequential edit data. While high-quality instruction data for code synthesis is scarce, edit data for synthesis is even scarcer. To fill this gap, we develop a synthetic data generation algorithm called LintSeq. This algorithm refactors programs into sequences of synthetic edits by using a linter to procedurally sample across interdependent lines of source code. Synthetic edits sampled with LintSeq reflect the syntax and semantics of their programming language. To test the algorithm, we use it to refactor a dataset of instruction + program pairs into instruction + program-diff-sequence tuples. Then, we fine-tune a series of smaller LMs ranging from 2.6B to 14B parameters on both the re-factored and original versions of this dataset. We perform comprehensive evaluations comparing edit sequence code LMs against baselines on HumanEval, MBPP(+), CodeContests, DS-1000, and BigCodeBench. We show that models fine-tuned to iteratively synthesize code match or outperform baselines on pass@1, and exhibit better scaling across higher pass@k as a function of total test-time FLOPs. Finally, we also pretrain our own tiny LMs for code understanding. We show that fine-tuning these models to synthesize code edit-by-edit results in strong performance on HumanEval and MBPP(+) compared to existing code language models of similar scale such as CodeT5+, AlphaCode, and Codex.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。