Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data. This study provides both quantitative and qualitative assessments of three public code language models when completing real-world code. We first developed an open-source IDE extension, Code4Me, for the online evaluation of the models. We collected real auto-completion usage data for over a year from more than 1200 users, resulting in over 600K valid completions. These models were then evaluated using six standard metrics across twelve programming languages. Next, we conducted a qualitative study of 1690 real-world completion requests to identify the reasons behind the poor model performance. A comparative analysis of the models' performance in online and offline settings was also performed, using benchmark synthetic datasets and two masking strategies. Our findings suggest that while developers utilize code completion across various languages, the best results are achieved for mainstream languages such as Python and Java. InCoder outperformed the other models across all programming languages, highlighting the significance of training data and objectives. Our study also revealed that offline evaluations do not accurately reflect real-world scenarios. Upon qualitative analysis of the model's predictions, we found that 66.3% of failures were due to the models' limitations, 24.4% occurred due to inappropriate model usage in a development context, and 9.3% were valid requests that developers overwrote. Given these findings, we propose several strategies to overcome the current limitations. These include refining training objectives, improving resilience to typographical errors, adopting hybrid approaches, and enhancing implementations and usability.
CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences
Delft University of Technology(代尔夫特理工大学)
代码完成是IDE的一个基本功能,但当前的自动完成器仅限于基于语法或基于NLP的单个标记完成。这两种方法都有明显的缺点:基于语法的自动完成在动态类型语言环境中受到限制,而基于NLP的自动完成程序很难理解程序设计语言的语义和开发人员的代码上下文。在这项工作中,我们提出了CodeFill,一个语言模型自动完成,结合了学习的结构和命名信息。使用并行Transformer架构和多任务学习,CodeFill使用源代码标记名称序列及其等效的AST标记类型。独特的是,CodeFill针对单标记和多标记(语句)预测进行了训练,这使其能够学习语法和命名元素之间的长期依赖关系。我们在两个数据集上训练CodeFill,分别由29 M和425 M行代码组成。为了使评估更加真实,我们开发了一种方法来自动推断源代码中的完成点。我们将CodeFill与四个基线和两个最先进的模型GPT-C和TravTrans+进行了比较。CodeFill在单令牌预测方面超过了所有基线(MRR:70.9% vs. 66.2%和67.8%),并且在多令牌预测方面优于最先进的模型(ROUGE-L:63.7% vs. 52.4%和59.2%,n=4个令牌)。我们公开发布我们的源代码和数据集。
在本文中,我们提出了一种新的端到端的AI辅助代码完成方法,称为Pythia。它生成方法和API建议的排名列表,可供软件开发人员在编辑时使用。该系统目前部署为Visual Studio Code IDE中Intellicode扩展的一部分。Pythia利用了最先进的大规模深度学习模型,这些模型是在从抽象语法树中提取的代码上下文上训练的。它被设计为在高吞吐量下工作,预测100个$ms$数量级的最佳匹配代码完成。我们描述了该系统的架构,执行比较基于频率的方法和基于调用的马尔可夫链语言模型,并讨论了轻量级客户端设备上的Pythia模型服务的挑战。在2700个Python开源软件GitHub存储库上获得的离线评估结果显示,对于项目内和跨项目设置,前5名的准确率为92%,超过基线模型20%。
Evaluating and Achieving Controllable Code Completion in Code LLM
University of Science and Technology of China(中国科学技术大学);Alibaba Group(阿里巴巴集团);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)
Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems
Singapore Management University(新加坡管理大学);Monash University(莫纳什大学);Key Laboratory of System Software (Chinese Academy of Sciences)(中国科学院系统软件重点实验室);State Key Laboratory of Computer Science(计算机科学国家重点实验室);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Nanjing Institute of Software Technology(南京软件技术学院);National University of Defense Technology(国防科技大学);University of Technology Sydney(悉尼科技大学);Beihang University(北京航空航天大学)
Teaching LLMs a Low-Resource Language: Enhancing Code Completion in Pharo
Graz University of Technology(格拉茨工业大学);Software Institute – USI(软件研究所 - 瑞士意大利大学);Università della Svizzera italiana(瑞士意大利大学);Univ. Lille, Inria, CNRS, Centrale Lille and Lifeware SA(里尔大学、法国国家信息与自动化研究院、法国国家科学研究中心、里尔中央理工学院和Lifeware公司);CIRAD(法国农业研究国际中心);CNRS, University of Bordeaux(法国国家科学研究中心、波尔多大学);Bordeaux INP, LaBRI, UMR5800(波尔多理工学院、拉布里实验室)
Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion
Nanjing University(南京大学);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);National Institute of Healthcare Data Science, Nanjing University(南京大学国家卫生数据科学研究院)
Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach
National Engineering Research Center for Big Data Technology and System(国家大数据技术与系统工程研究中心);Huazhong University of Science and Technology(华中科技大学);Chongqing University(重庆大学);University of Leigh(利哈伊大学)
CodeRAG: Finding Relevant and Necessary Knowledge for Retrieval-Augmented Repository-Level Code Completion
Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China(教育部多媒体可信感知与高效计算重点实验室);Xiamen University(厦门大学);Ant Group(蚂蚁集团)
Retrieval-augmented code completion for local projects using large language models
Faculty of Computer and Information Science, University of Ljubljana(计算机与信息科学学院,卢布尔雅那大学);Faculty of Mathematics and Physics, University of Ljubljana(数学与物理学院,卢布尔雅那大学)