arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.16197cs.SEcs.LGcs.PL
收藏

用于代码补全的语言模型:一项实用评估

Language Models for Code Completion: A Practical Evaluation

Delft University of Technology(代尔夫特理工大学)

Maliheh Izadi, Jonathan Katzy, Tim van Dam, Marc Otten, Razvan Mihai Popescu, Arie van Deursen

AI总结:

本文通过Code4Me扩展收集超60万条真实补全数据,定量与定性评估三个代码语言模型,发现InCoder表现最佳且离线评估失真,并提出改进训练目标与混合方法等策略。

摘要:

基于Transformer的自动代码补全语言模型迄今已展现出巨大潜力,然而对这些模型的评估很少使用真实数据。本研究对三个公开代码语言模型在补全真实世界代码时的表现进行了定量和定性评估。我们首先开发了一个开源IDE扩展Code4Me,用于模型的在线评估。我们收集了来自1200多名用户超过一年的真实自动补全使用数据,得到超过60万条有效补全。随后使用十二种编程语言上的六项标准指标对这些模型进行了评估。接下来,我们对1690个真实世界补全请求进行了定性研究,以识别模型表现不佳背后的原因。我们还使用基准合成数据集和两种掩码策略,对模型在在线和离线设置下的性能进行了比较分析。我们的发现表明,尽管开发者在多种语言中使用代码补全,但最佳结果出现在Python和Java等主流语言上。InCoder在所有编程语言上都优于其他模型,凸显了训练数据和训练目标的重要性。我们的研究还揭示,离线评估并不能准确反映真实世界场景。在对模型预测进行定性分析后,我们发现66.3%的失败源于模型自身的局限性,24.4%是由于在开发环境中对模型的不当使用,9.3%是开发者覆盖掉的有效请求。基于这些发现,我们提出了若干策略来克服当前局限,包括改进训练目标、提高对拼写错误的鲁棒性、采用混合方法,以及改进实现和可用性。

英文摘要:

Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data. This study provides both quantitative and qualitative assessments of three public code language models when completing real-world code. We first developed an open-source IDE extension, Code4Me, for the online evaluation of the models. We collected real auto-completion usage data for over a year from more than 1200 users, resulting in over 600K valid completions. These models were then evaluated using six standard metrics across twelve programming languages. Next, we conducted a qualitative study of 1690 real-world completion requests to identify the reasons behind the poor model performance. A comparative analysis of the models' performance in online and offline settings was also performed, using benchmark synthetic datasets and two masking strategies. Our findings suggest that while developers utilize code completion across various languages, the best results are achieved for mainstream languages such as Python and Java. InCoder outperformed the other models across all programming languages, highlighting the significance of training data and objectives. Our study also revealed that offline evaluations do not accurately reflect real-world scenarios. Upon qualitative analysis of the model's predictions, we found that 66.3% of failures were due to the models' limitations, 24.4% occurred due to inappropriate model usage in a development context, and 9.3% were valid requests that developers overwrote. Given these findings, we propose several strategies to overcome the current limitations. These include refining training objectives, improving resilience to typographical errors, adopting hybrid approaches, and enhancing implementations and usability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑