生成概率并不足够:AI 代码补全中的不确定性高亮
Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
AI总结:
本研究通过混合方法实验,发现突出显示编辑预测可能性高的标记比生成概率高亮更能提升程序员在AI代码补全中的速度和准确性,且更受偏好。
摘要:
大规模生成模型推动了 AI 驱动的代码补全工具的发展,以协助程序员编写代码。然而,与其他 AI 驱动的工具类似,AI 代码补全并不总是准确的,如果人类程序员未能正确检测和纠正,可能会将错误甚至安全漏洞引入代码中。为帮助程序员识别潜在错误,已提出并实现的一种技术是突出显示不确定的标记。然而,尚无实证研究探讨该技术的有效性,也未研究在生成模型背景下不同且尚未达成共识的不确定性概念。我们探讨了这样一个问题:在程序员与 AI 驱动的代码补全工具协作时,传达不确定性信息是否能使他们更快、更准确地生成代码,如果是,哪种不确定性度量最符合程序员的需求。通过一项包含 30 名程序员的混合方法研究,我们比较了三种条件:仅提供 AI 系统的代码补全、突出显示底层生成模型生成可能性最低的标记,以及突出显示程序员编辑预测可能性最高的标记。我们发现,突出显示编辑预测可能性最高的标记能带来更快的任务完成速度和更有针对性的编辑,并且研究参与者主观上更偏好这种方式。相比之下,根据生成概率突出显示标记与无高亮的基线相比没有任何优势。我们进一步探索了在 AI 驱动的代码补全工具中传达不确定性的设计空间,发现程序员偏好那些细粒度、信息丰富、可解释且不过度令人负担的高亮方式。
英文摘要:
Large-scale generative models enabled the development of AI-powered code completion tools to assist programmers in writing code. However, much like other AI-powered tools, AI-powered code completions are not always accurate, potentially introducing bugs or even security vulnerabilities into code if not properly detected and corrected by a human programmer. One technique that has been proposed and implemented to help programmers identify potential errors is to highlight uncertain tokens. However, there have been no empirical studies exploring the effectiveness of this technique -- nor investigating the different and not-yet-agreed-upon notions of uncertainty in the context of generative models. We explore the question of whether conveying information about uncertainty enables programmers to more quickly and accurately produce code when collaborating with an AI-powered code completion tool, and if so, what measure of uncertainty best fits programmers' needs. Through a mixed-methods study with 30 programmers, we compare three conditions: providing the AI system's code completion alone, highlighting tokens with the lowest likelihood of being generated by the underlying generative model, and highlighting tokens with the highest predicted likelihood of being edited by a programmer. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits, and is subjectively preferred by study participants. In contrast, highlighting tokens according to their probability of being generated does not provide any benefit over the baseline with no highlighting. We further explore the design space of how to convey uncertainty in AI-powered code completion tools, and find that programmers prefer highlights that are granular, informative, interpretable, and not overwhelming.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。