深入探究代码变更学习中的参数高效微调:一项实证研究
Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study
AI总结:
该实证研究对比了参数高效微调(PEFT)与全模型微调(FMFT)在代码变更任务上的性能,发现适配器调优和低秩适配在即时缺陷预测和提交消息生成上表现优越,尤其在跨语言和低资源场景下。
摘要:
与全模型微调(FMFT)相比,参数高效微调(PEFT)在若干代码理解任务(如代码摘要和代码搜索)中展现出更优的性能和更低的计算开销。这一优势可归因于PEFT通过仅更新少量参数,能够缓解预训练语言模型(PLMs)的灾难性遗忘问题。因此,PEFT有效地利用了预训练的通用于下游任务的知识。然而,现有研究主要涉及静态代码理解,这与近期PLMs的预训练范式一致并促进知识迁移,但未考虑动态代码变更。因此,PEFT在代码变更相关任务的任务特定适配中是否优于FMFT仍不清楚。为解决此问题,我们考察了两种流行的PEFT方法,即适配器调优(AT)和低秩适配(LoRA),并在五个流行的PLMs上将其性能与FMFT进行比较。具体而言,我们在两个广泛研究的代码变更相关任务上评估其性能:即时缺陷预测(JIT-DP)和提交消息生成(CMG)。结果表明,与FMFT及其他最先进(SOTA)方法相比,AT和LoRA在JIT-DP上均取得SOTA结果,在CMG上表现出相当的性能。此外,AT和LoRA在跨语言和低资源场景中展现出优越性。我们还进行了三项探测任务,从静态和动态角度解释PEFT技术在JIT-DP和CMG任务上的有效性。研究表明,PEFT,特别是通过使用AT和LoRA,在代码变更相关任务中提供了有前景的优势,在某些方面超越了FMFT。
英文摘要:
Compared to Full-Model Fine-Tuning (FMFT), Parameter Efficient Fine-Tuning (PEFT) has demonstrated superior performance and lower computational overhead in several code understanding tasks, such as code summarization and code search. This advantage can be attributed to PEFT's ability to alleviate the catastrophic forgetting issue of Pre-trained Language Models (PLMs) by updating only a small number of parameters. As a result, PEFT effectively harnesses the pre-trained general-purpose knowledge for downstream tasks. However, existing studies primarily involve static code comprehension, aligning with the pre-training paradigm of recent PLMs and facilitating knowledge transfer, but they do not account for dynamic code changes. Thus, it remains unclear whether PEFT outperforms FMFT in task-specific adaptation for code-change-related tasks. To address this question, we examine two prevalent PEFT methods, namely Adapter Tuning (AT) and Low-Rank Adaptation (LoRA), and compare their performance with FMFT on five popular PLMs. Specifically, we evaluate their performance on two widely-studied code-change-related tasks: Just-In-Time Defect Prediction (JIT-DP) and Commit Message Generation (CMG). The results demonstrate that both AT and LoRA achieve state-of-the-art (SOTA) results in JIT-DP and exhibit comparable performances in CMG when compared to FMFT and other SOTA approaches. Furthermore, AT and LoRA exhibit superiority in cross-lingual and low-resource scenarios. We also conduct three probing tasks to explain the efficacy of PEFT techniques on JIT-DP and CMG tasks from both static and dynamic perspectives. The study indicates that PEFT, particularly through the use of AT and LoRA, offers promising advantages in code-change-related tasks, surpassing FMFT in certain aspects.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。