arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.05026cs.SEcs.LG
收藏

面向本地项目的大语言模型检索增强代码补全

Retrieval-augmented code completion for local projects using large language models

Faculty of Computer and Information Science, University of Ljubljana(计算机与信息科学学院,卢布尔雅那大学);Faculty of Mathematics and Physics, University of Ljubljana(数学与物理学院,卢布尔雅那大学)

Marko Hostnik, Marko Robnik-Šikonja

AI总结:

针对大语言模型代码补全的隐私与计算问题,提出基于本地项目的上下文内检索增强生成方法,经实验验证其性能优于同规模RETRO架构,显著提升代码补全效果。

摘要:

大语言模型(LLM)在软件开发者中的应用正日益广泛。然而,商业解决方案及大语言模型的使用存在隐私与计算需求方面的问题。本研究聚焦于采用参数量为1.6亿、适合本地运行且可通过本地项目检索进行增强的相对轻量高效的大语言模型。我们在开源Python文件上训练了两个基于transformer的开源模型——生成式GPT-2和适配检索的RETRO,并对二者进行实证比较,证实了基于嵌入的检索的优势。此外,我们通过上下文内检索增强生成(RAG)提升模型性能,该方法利用词元的Jaccard相似度检索代码片段。我们在更大规模的模型上评估了上下文内RAG,发现尽管该方法原理简单,但比采用RETRO架构更适用。实验结果表明,上下文内RAG将代码补全基线性能提升了26%以上,而RETRO较同等规模的GPT-2基线提升了12%。我们强调,恰当的词元化对充分发挥大语言模型在代码补全中的潜力具有关键作用。

英文摘要:

The use of large language models (LLMs) is becoming increasingly widespread among software developers. However, privacy and computational requirements are problematic with commercial solutions and the use of LLMs. In this work, we focus on using relatively small and efficient LLMs with 160M parameters that are suitable for local execution and augmentation with retrieval from local projects. We train two open transformer-based models, the generative GPT-2 and the retrieval-adapted RETRO, on open-source Python files, and empirically compare them, confirming the benefits of embedding-based retrieval. Furthermore, we improve our models' performance with In-context retrieval-augmented generation (RAG), which retrieves code snippets using the Jaccard similarity of tokens. We evaluate In-context RAG on larger models and determine that, despite its simplicity, the approach is more suitable than using the RETRO architecture. Experimental results indicate that In-context RAG improves the code completion baseline by over 26%, while RETRO improves over the similarly sized GPT-2 baseline by 12%. We highlight the key role of proper tokenization in achieving the full potential of LLMs in code completion.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑