IRCoCo:面向代码补全的即时奖励引导深度强化学习
IRCoCo: Immediate Rewards-Guided Deep Reinforcement Learning for Code Completion
AI总结:
针对代码补全中曝光偏差和动态上下文适应难题,本文提出基于深度强化学习的IRCoCo框架,通过即时奖励检测上下文变化并精细优化语言模型,实验证明其性能优于SFT和其他DRL基线。
摘要:
代码补全旨在通过基于当前编程上下文预测潜在代码来提升编程效率。近年来,预训练语言模型(LMs)已成为该领域的主流方法。已有多种方法被提出,利用监督微调(SFT)技术对语言模型进行微调以完成代码补全任务。然而,这些模型固有的曝光偏差会导致错误在序列补全的早期不断累积,进而在后续补全中引发更多错误。为解决这一问题,深度强化学习(DRL)作为微调代码补全语言模型的替代技术,能够提升模型的泛化能力和整体性能。尽管如此,将基于DRL的策略集成到代码补全中面临两大挑战:1)代码上下文的动态特性要求补全模型快速适应变化,这对专注于最终代码状态延迟奖励的传统DRL策略构成困难。2)部分代码的正确性难以评估,因此基于奖励重分配的策略无法适用于代码补全。为应对这些挑战,我们提出了IRCoCo,一个专门针对代码补全的基于DRL的微调框架。该框架旨在提供即时奖励作为反馈,以检测代码补全过程中由持续编辑引起的动态上下文变化。借助即时反馈,微调后的语言模型能够更精确地理解当前上下文,从而有效调整语言模型并以更精细的方式优化代码补全。实验结果表明,使用IRCoCo微调预训练语言模型能在代码补全任务中带来显著提升,性能优于基于SFT的方法以及其他基于DRL的基线方法。
英文摘要:
Code completion aims to enhance programming productivity by predicting potential code based on the current programming context. Recently, pretrained language models (LMs) have become prominent in this field. Various approaches have been proposed to fine-tune LMs using supervised fine-tuning (SFT) techniques for code completion. However, the inherent exposure bias of these models can cause errors to accumulate early in the sequence completion, leading to even more errors in subsequent completions. To address this problem, deep reinforcement learning (DRL) is an alternative technique for fine-tuning LMs for code completion, which can improve the generalization capabilities and overall performance. Nevertheless, integrating DRL-based strategies into code completion faces two major challenges: 1) The dynamic nature of the code context requires the completion model to quickly adapt to changes, which poses difficulties for conventional DRL strategies that focus on delayed rewarding of the final code state. 2) It is difficult to evaluate the correctness of partial code, thus the reward redistribution-based strategies cannot be adapted to code completion. To tackle these challenges, we propose IRCoCo, a code completion-specific DRL-based fine-tuning framework. This framework is designed to provide immediate rewards as feedback for detecting dynamic context changes arising from continuous edits during code completion. With the aid of immediate feedback, the fine-tuned LM can gain a more precise understanding of the current context, thereby enabling effective adjustment of the LM and optimizing code completion in a more refined manner. Experimental results demonstrate that fine-tuning pretrained LMs with IRCoCo leads to significant improvements in the code completion task, outperforming both SFT-based and other DRL-based baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。