arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.03519cs.SEcs.AI
收藏

RepoMasterEval：通过真实世界代码库评估代码补全

RepoMasterEval: Evaluating Code Completion via Real-World Repositories

Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳校区）

Qinyun Wu, Chao Peng, Pengfei Gao, Ruida Hu, Haoyu Gan, Bo Jiang, Jinhe Tang, Zhiwen Deng, Zhanming Guan, Cuiyun Gao, Xia Liu, Ping Yang

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 RepoMasterEval 基准，基于真实代码库评估代码补全模型，利用变异测试增强测试套件，实证表明其能准确反映模型在实际场景中的性能差异并提供有效训练反馈。

摘要：

随着软件开发中对自动化代码补全工具的依赖日益增加，对全面评估基准的需求变得至关重要。现有基准更多地关注函数和类级别的代码补全，通过提供文本描述来提示模型。相比之下，这种描述性提示在实际开发中通常不可用，并且代码补全可能发生在更广泛的场景中，例如在函数或代码块的中间。这些限制使得现有的评估基准与代码补全工具的实际场景不能很好地对齐。在本文中，我们提出了 RepoMasterEval，这是一种用于评估代码补全模型的新颖基准，它是从真实世界的代码库中构建的。每个基准数据都是通过从带有现有测试套件的一个源代码文件中掩码代码片段（真实值）来生成的。为了提高模型生成代码的测试准确性，我们采用变异测试来衡量测试用例的有效性，并为那些变异得分较低的测试套件手动编写了新的测试用例。我们对 10 个最先进的模型进行的实证评估表明，测试增强对于提高基准的准确性至关重要，并且 RepoMasterEval 能够报告模型在真实世界场景中的性能差异。RepoMasterEval 的部署还表明，该基准在模型训练期间有助于提供准确的反馈，并且其得分与模型在实际中的性能高度相关。

英文摘要：

With the growing reliance on automated code completion tools in software development, the need for comprehensive evaluation benchmarks has become critical. Existing benchmarks focus more on code completion in function and class level by providing text descriptions to prompt the model. By contrast, such descriptive prompt is commonly unavailable in real development and code completion can occur in wider range of situations such as in the middle of a function or a code block. These limitations makes existing evaluation benchmarks poorly align with the practical scenarios of code completion tools. In this paper, we propose RepoMasterEval, a novel benchmark for evaluating code completion models constructed from real-world repositories. Each benchmark datum is generated by masking a code snippet (ground truth) from one source code file with existing test suites. To improve test accuracy of model generated code, we employ mutation testing to measure the effectiveness of the test cases and we manually crafted new test cases for those test suites with low mutation score. Our empirical evaluation on 10 state-of-the-art models shows that test argumentation is critical in improving the accuracy of the benchmark and RepoMasterEval is able to report variance in model performance in real-world scenarios. The deployment of RepoMasterEval also revealed that the benchmark is useful to give accurate feedback during model training and the score is in high correlation with the model's performance in practice.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑