Retrieval-augmented code completion for local projects using large language models
Faculty of Computer and Information Science, University of Ljubljana(计算机与信息科学学院,卢布尔雅那大学);Faculty of Mathematics and Physics, University of Ljubljana(数学与物理学院,卢布尔雅那大学)
Marko Hostnik, Marko Robnik-Šikonja
备注:
30 pages, 15 figures; Accepted manuscript for Expert Systems with Applications
The use of large language models (LLMs) is becoming increasingly widespread among software developers. However, privacy and computational requirements are problematic with commercial solutions and the use of LLMs. In this work, we focus on using relatively small and efficient LLMs with 160M parameters that are suitable for local execution and augmentation with retrieval from local projects. We train two open transformer-based models, the generative GPT-2 and the retrieval-adapted RETRO, on open-source Python files, and empirically compare them, confirming the benefits of embedding-based retrieval. Furthermore, we improve our models' performance with In-context retrieval-augmented generation (RAG), which retrieves code snippets using the Jaccard similarity of tokens. We evaluate In-context RAG on larger models and determine that, despite its simplicity, the approach is more suitable than using the RETRO architecture. Experimental results indicate that In-context RAG improves the code completion baseline by over 26%, while RETRO improves over the similarly sized GPT-2 baseline by 12%. We highlight the key role of proper tokenization in achieving the full potential of LLMs in code completion.
CodeRAG: Finding Relevant and Necessary Knowledge for Retrieval-Augmented Repository-Level Code Completion
Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China(教育部多媒体可信感知与高效计算重点实验室);Xiamen University(厦门大学);Ant Group(蚂蚁集团)
RepoShapley: Shapley-Enhanced Context Filtering for Repository-Level Code Completion
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)理工学院);Sun Yat-sen University(中山大学);University of California, San Diego(加利福尼亚大学圣地亚哥分校);Hangzhou Dianzi University(杭州电子科技大学);Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)(深圳未来网络智能研究所);Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)(香港中文大学(深圳)广东省未来网络智能重点实验室)
Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion
Nanjing University(南京大学);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);National Institute of Healthcare Data Science, Nanjing University(南京大学国家卫生数据科学研究院)