arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.01554cs.SE
收藏

FT2Ra:一种受微调启发的检索增强代码补全方法

FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion

Qi Guo, Xiaohong Li, Xiaofei Xie, Shangqing Liu, Ze Tang, Ruitao Feng, Junjie Wang, Jidong Ge, Lei Bu

AI总结:

FT2Ra 通过理论分析揭示 delta logits 对微调的关键作用,并据此设计了一种带学习率和多轮次检索的检索增强方法,无需真实微调即可在词元级和行级代码补全中显著提升准确率与 Exact Match 性能。

摘要:

代码预训练模型的兴起显著提升了各种编码任务的表现,例如代码补全,以及 GitHub Copilot 等工具。然而,这些模型尤其是大型模型的庞大体积,给针对特定下游任务进行微调带来了重大挑战。作为一种替代方案,基于检索的方法已成为一种有前景的解决方案,无需微调即可增强模型预测。尽管具有潜力,但一个重大挑战在于这些方法的设计往往依赖启发式规则,留下了关键问题:应该存储或检索什么信息,以及如何插值这些信息以增强预测。为应对这一挑战,我们首先对微调过程进行了理论分析,强调了 delta logits 作为改善模型预测催化剂的重要性。基于这一见解,我们开发了一种新颖的基于检索的方法 FT2Ra,旨在模拟真正的微调。虽然 FT2Ra 采用基于检索的机制,但它独特地采用了具有学习率和多轮次检索的范式,这与微调类似。在相对较简单的词元级补全任务中,FT2Ra 在 UniXcoder 上相比最佳基线方法实现了 4.29% 的准确率提升。在更具挑战性的行级补全任务中,我们观察到 Exact Match(EM)性能提升超过两倍,表明我们理论分析的显著优势。值得注意的是,即使在没有实际微调的情况下运行,FT2Ra 也表现出与经过真实微调的模型相当的竞争力。

英文摘要:

The rise of code pre-trained models has significantly enhanced various coding tasks, such as code completion, and tools like GitHub Copilot. However, the substantial size of these models, especially large models, poses a significant challenge when it comes to fine-tuning them for specific downstream tasks. As an alternative approach, retrieval-based methods have emerged as a promising solution, augmenting model predictions without the need for fine-tuning. Despite their potential, a significant challenge is that the designs of these methods often rely on heuristics, leaving critical questions about what information should be stored or retrieved and how to interpolate such information for augmenting predictions. To tackle this challenge, we first perform a theoretical analysis of the fine-tuning process, highlighting the importance of delta logits as a catalyst for improving model predictions. Building on this insight, we develop a novel retrieval-based method, FT2Ra, which aims to mimic genuine fine-tuning. While FT2Ra adopts a retrieval-based mechanism, it uniquely adopts a paradigm with a learning rate and multi-epoch retrievals, which is similar to fine-tuning.In token-level completion, which represents a relatively easier task, FT2Ra achieves a 4.29% improvement in accuracy compared to the best baseline method on UniXcoder. In the more challenging line-level completion task, we observe a substantial more than twice increase in Exact Match (EM) performance, indicating the significant advantages of our theoretical analysis. Notably, even when operating without actual fine-tuning, FT2Ra exhibits competitive performance compared to the models with real fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑