代码大语言模型在补全含有潜在错误的代码时失败
Large Language Models of Code Fail at Completing Code with Potential Bugs
AI总结：
本文提出并研究有错误代码补全问题，构建合成与真实错误数据集，发现潜在错误显著降低代码大语言模型的补全性能，并验证事后缓解方法仍存在明显不足。
摘要：
代码大语言模型（Code-LLMs）最近为代码补全带来了巨大进步，代码补全是编程辅助和代码智能的一项基本功能。然而，大多数现有工作忽略了生成时代码上下文中可能存在错误的情况，而这些错误在软件开发中不可避免。因此，受实时代码建议这一现实场景的启发——其中代码上下文包含潜在错误，即可能成为完成程序中错误的“反模式”——我们引入并研究了“有错误代码补全”问题。为了系统地研究该任务，我们引入了两个数据集：一个包含由改变语义的运算符更改派生的合成错误（buggy-HumanEval），另一个包含来自用户提交的编程题解答的真实错误（buggy-FixEval）。我们发现，潜在错误的存在显著降低了高性能代码大语言模型的生成性能。例如，当上下文中存在单个潜在错误时，CODEGEN-2B-MONO 在 buggy-HumanEval 测试用例上的通过率下降超过 50%。最后，我们研究了几种事后缓解潜在错误不利影响的方法，并发现缓解后的性能仍然存在显著差距。
英文摘要：
Large language models of code (Code-LLMs) have recently brought tremendous advances to code completion, a fundamental feature of programming assistance and code intelligence. However, most existing works ignore the possible presence of bugs in the code context for generation, which are inevitable in software development. Therefore, we introduce and study the buggy-code completion problem, inspired by the realistic scenario of real-time code suggestion where the code context contains potential bugs -- anti-patterns that can become bugs in the completed program. To systematically study the task, we introduce two datasets: one with synthetic bugs derived from semantics-altering operator changes (buggy-HumanEval) and one with realistic bugs derived from user submissions to coding problems (buggy-FixEval). We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs. For instance, the passing rates of CODEGEN-2B-MONO on test cases of buggy-HumanEval drop more than 50% given a single potential bug in the context. Finally, we investigate several post-hoc methods for mitigating the adverse effect of potential bugs and find that there remains a significant gap in post-mitigation performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
