当神经代码补全模型审时度势:通过动态模型推理实现更廉价、更快速的补全
When Neural Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inference
AI总结:
本文针对大规模代码补全模型计算成本高的问题,提出一种动态推理方法,通过决策机制提前停止错误生成并跳过冗余层,在几乎不损失精度的情况下实现11.2%的加速。
摘要:
借助大语言模型的最新进展,现代神经代码补全模型已展现出生成高精度代码建议的能力。然而,其庞大的规模在计算成本和环境影响方面带来了挑战,阻碍了它们在实际场景中的广泛应用。动态推理作为一种有前景的解决方案应运而生,因为它在保持模型性能的同时,在推理过程中分配最少的计算量。在本研究中,我们在代码补全的背景下探索动态推理。首先,我们对GPT-2进行了实证研究,重点关注中间层在代码补全中的推理能力。我们发现54.4%的token仅使用第一层就能被准确生成,这表明存在显著的计算节省潜力。此外,尽管使用了所有层,模型仍然无法正确预测14.5%的token,并且从这些错误预测继续生成的后续补全很少被认为是有帮助的,其接受率仅为4.2%。这些发现促使我们在代码补全中探索动态推理,并启发我们通过一种决策机制来增强它,该机制可以阻止错误代码的生成。因此,我们提出了一种专门为代码补全模型量身定制的新型动态推理方法。该方法不仅旨在以大幅减少的计算量产生正确的预测,而且还能主动防止错误的预测。我们广泛的评估表明,该方法平均可以跳过模型中16层中的1.7层,从而实现11.2%的加速,而ROUGE-L仅略微下降1.1%。
英文摘要:
Leveraging recent advancements in large language models, modern neural code completion models have demonstrated the capability to generate highly accurate code suggestions. However, their massive size poses challenges in terms of computational costs and environmental impact, hindering their widespread adoption in practical scenarios. Dynamic inference emerges as a promising solution, as it allocates minimal computation during inference while maintaining the model's performance. In this research, we explore dynamic inference within the context of code completion. Initially, we conducted an empirical investigation on GPT-2, focusing on the inference capabilities of intermediate layers for code completion. We found that 54.4% of tokens can be accurately generated using just the first layer, signifying significant computational savings potential. Moreover, despite using all layers, the model still fails to predict 14.5% of tokens correctly, and the subsequent completions continued from them are rarely considered helpful, with only a 4.2% Acceptance Rate. These findings motivate our exploration of dynamic inference in code completion and inspire us to enhance it with a decision-making mechanism that stops the generation of incorrect code. We thus propose a novel dynamic inference method specifically tailored for code completion models. This method aims not only to produce correct predictions with largely reduced computation but also to prevent incorrect predictions proactively. Our extensive evaluation shows that it can averagely skip 1.7 layers out of 16 layers in the models, leading to an 11.2% speedup with only a marginal 1.1% reduction in ROUGE-L.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。