The rise of code pre-trained models has significantly enhanced various coding tasks, such as code completion, and tools like GitHub Copilot. However, the substantial size of these models, especially large models, poses a significant challenge when it comes to fine-tuning them for specific downstream tasks. As an alternative approach, retrieval-based methods have emerged as a promising solution, augmenting model predictions without the need for fine-tuning. Despite their potential, a significant challenge is that the designs of these methods often rely on heuristics, leaving critical questions about what information should be stored or retrieved and how to interpolate such information for augmenting predictions.
To tackle this challenge, we first perform a theoretical analysis of the fine-tuning process, highlighting the importance of delta logits as a catalyst for improving model predictions. Building on this insight, we develop a novel retrieval-based method, FT2Ra, which aims to mimic genuine fine-tuning. While FT2Ra adopts a retrieval-based mechanism, it uniquely adopts a paradigm with a learning rate and multi-epoch retrievals, which is similar to fine-tuning.In token-level completion, which represents a relatively easier task, FT2Ra achieves a 4.29% improvement in accuracy compared to the best baseline method on UniXcoder. In the more challenging line-level completion task, we observe a substantial more than twice increase in Exact Match (EM) performance, indicating the significant advantages of our theoretical analysis. Notably, even when operating without actual fine-tuning, FT2Ra exhibits competitive performance compared to the models with real fine-tuning.
RepoShapley: Shapley-Enhanced Context Filtering for Repository-Level Code Completion
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)理工学院);Sun Yat-sen University(中山大学);University of California, San Diego(加利福尼亚大学圣地亚哥分校);Hangzhou Dianzi University(杭州电子科技大学);Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)(深圳未来网络智能研究所);Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)(香港中文大学(深圳)广东省未来网络智能重点实验室)
Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion
Nanjing University(南京大学);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);National Institute of Healthcare Data Science, Nanjing University(南京大学国家卫生数据科学研究院)
Retrieval-augmented code completion for local projects using large language models
Faculty of Computer and Information Science, University of Ljubljana(计算机与信息科学学院,卢布尔雅那大学);Faculty of Mathematics and Physics, University of Ljubljana(数学与物理学院,卢布尔雅那大学)
CodeRAG: Finding Relevant and Necessary Knowledge for Retrieval-Augmented Repository-Level Code Completion
Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China(教育部多媒体可信感知与高效计算重点实验室);Xiamen University(厦门大学);Ant Group(蚂蚁集团)