arXiv 2209.059482022-09cs.SEcs.AI

不要补全它！防止无益的代码补全，以构建高效且可持续的神经代码补全系统

Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems

Singapore Management University（新加坡管理大学）；Monash University（莫纳什大学）；Key Laboratory of System Software (Chinese Academy of Sciences)（中国科学院系统软件重点实验室）；State Key Laboratory of Computer Science（计算机科学国家重点实验室）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Nanjing Institute of Software Technology（南京软件技术学院）；National University of Defense Technology（国防科技大学）；University of Technology Sydney（悉尼科技大学）；Beihang University（北京航空航天大学）

Zhensu Sun, Xiaoning Du, Fu Song, Shangwen Wang, Mingze Ni, Li Li, David Lo

备注：: Accepted by TOSEM

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对大型代码模型约70%补全未被接受且浪费算力的问题，本文识别低回报提示模式，并提出基于质量预测的早期拒绝机制，五类估计器实验显示可拒绝20%请求且精确率达97.4%。

摘要：

当前，大型预训练语言模型被广泛应用于神经代码补全系统。尽管大型代码模型显著优于较小的同类模型，但 GitHub Copilot 显示的代码补全中约有 70% 未被开发者接受。这些补全经过审查却未被接受，因此对开发者生产力的帮助相当有限，甚至可能反过来加重开发者的工作负担，因为在最先进的代码补全系统中，一旦启用服务，代码补全就会在开发者输入时自动、主动地生成。更糟糕的是，考虑到大型代码模型的高昂成本，这会造成巨大的计算资源和能源浪费，严重违背 AI 技术的可持续发展原则。然而，在神经代码补全研究领域，这种浪费从未被认识到，更不用说得到有效解决。因此，迫切需要以成本友好的方式防止此类无益的代码补全发生。为填补这一重要空白，我们首先研究了无益代码补全对应的提示，将其称为“低回报提示”。我们通过实证识别出低回报提示中的四种可观察模式；每种模式都缺乏必要信息，因此难以仅通过提升模型准确性来解决。这证明了基于提示本身识别此类低回报提示的可行性。受这一发现启发，我们提出一种早期拒绝机制，通过预测代码补全质量来拒绝低回报提示。被估计将产生无益代码补全的提示不会发送给模型。此外，我们研究了五类估计器，以证明该机制的可行性。实验结果表明，该估计器能够拒绝 20% 的代码补全请求，精确率达到 97.4%。

英文摘要：

Currently, large pre-trained language models are widely applied in neural code completion systems. Though large code models significantly outperform their smaller counterparts, around 70\% of displayed code completions from Github Copilot are not accepted by developers. Being reviewed but not accepted, their help to developer productivity is considerably limited and may conversely aggravate the workload of developers, as the code completions are automatically and actively generated in state-of-the-art code completion systems as developers type out once the service is enabled. Even worse, considering the high cost of the large code models, it is a huge waste of computing resources and energy, which severely goes against the sustainable development principle of AI technologies. However, such waste has never been realized, not to mention effectively addressed, in the research community for neural code completion. Hence, preventing such unhelpful code completions from happening in a cost-friendly way is of urgent need. To fill this significant gap, we first investigate the prompts of unhelpful code completions, called "low-return prompts". We empirically identify four observable patterns in low-return prompts, each lacking necessary information, making it difficult to address through enhancements to the model's accuracy alone. This demonstrates the feasibility of identifying such low-return prompts based on the prompts themselves. Motivated by this finding, we propose an early-rejection mechanism to turn down low-return prompts by foretelling the code completion qualities. The prompts that are estimated to receive unhelpful code completions will not be sent to the model. Furthermore, we investigated five types of estimators to demonstrate the feasibility of the mechanism. The experimental results show that the estimator can reject 20% of code completion requests with a 97.4% Precision.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。