arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.13697cs.SEcs.LG
收藏

项目级代码补全的预训练研究

On Pretraining for Project-Level Code Completion

JetBrains Research(JetBrains 研究)

Maksim Sapronov, Evgeniy Glukhov

AI总结:

本研究探讨仓库级预训练策略对OpenCoder模型代码补全的影响,通过扩展上下文窗口和调整RoPE参数,在较小数据集上达到与大型模型相当的性能,并验证了文件级训练的简单高效性。

摘要:

仓库级预训练通常用于使大型代码语言模型能够利用代码库范围的上下文。这增强了它们生成准确且上下文感知的代码补全的能力。在这项工作中,我们研究了不同的仓库处理策略如何影响OpenCoder(一个1.5B参数的模型)中的上下文学习。我们通过在额外的1B个精选仓库级数据令牌上进行训练,将其上下文窗口从4,096扩展到16,384个令牌。尽管依赖的数据集比竞争模型(通常使用数千亿个令牌)更小,我们的模型在Long Code Arena基准上取得了相当的性能。我们发现各种仓库处理技术产生了类似强劲的结果,主要收益来自于适应新的旋转位置嵌入(RoPE)缩放参数。最后,我们展示了在原始序列长度下更简单的文件级训练方法仍然非常有效,为数据和计算资源更受限的设置打开了仓库级代码补全研究的大门。

英文摘要:

Repository-level pretraining is commonly used to enable large language models for code to leverage codebase-wide context. This enhances their ability to generate accurate and context-aware code completions. In this work, we investigate how different repository-processing strategies affect in-context learning in OpenCoder, a 1.5B-parameter model. We extend its context window from 4,096 to 16,384 tokens by training on additional 1B tokens of curated repository-level data. Despite relying on a smaller dataset than competing models (which often use hundreds of billions of tokens), our model achieves comparable performance on the Long Code Arena benchmark. We find that various repository-processing techniques yield similarly strong results, with the primary gain coming from adapting to a new rotary positional embedding (RoPE) scaling parameter. Finally, we show that a simpler file-level training approach at the original sequence length remains highly effective, opening up repository-level code completion research to settings with more constrained data and compute resources.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑