arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.15822cs.SEcs.AI
收藏

所有程序设计语言的一个适配器?用于代码搜索和摘要的适配器调优

One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization

National University of Defense Technology(国防科技大学);Hunan University(湖南大学)

Deze Wang, Boxing Chen, Shanshan Li, Wei Luo, Shaoliang Peng, Wei Dong, Xiangke Liao

摘要:

由于预先训练的模型使许多代码智能任务自动化,因此广泛使用的范例是针对每种程序设计语言在任务数据集上微调模型。最近的一项研究表明,多语言微调有利于一系列任务和模型。然而,我们发现,多语言微调导致最近的模型UniXcoder和CodeT5的性能下降。为了缓解多语言模型中潜在的灾难性遗忘问题,我们修复了所有预先训练的模型参数,插入参数有效的结构适配器,并对其进行微调。与每种程序设计语言的全模型微调相比,适配器调优仅更新了整体参数的0.6%,在代码搜索和摘要任务上产生了一致的改进,实现了最先进的结果。此外,我们的实验表明其有效性,在跨语言和低资源的情况下。每种程序设计语言200个样本的多语言微调接近于在代码摘要上使用整个数据集进行微调的结果。我们对三个探测任务的实验表明,适配器调整显着优于全模型微调,并有效地克服了灾难性遗忘。

英文摘要:

As pre-trained models automate many code intelligence tasks, a widely used paradigm is to fine-tune a model on the task dataset for each programming language. A recent study reported that multilingual fine-tuning benefits a range of tasks and models. However, we find that multilingual fine-tuning leads to performance degradation on recent models UniXcoder and CodeT5. To alleviate the potentially catastrophic forgetting issue in multilingual models, we fix all pre-trained model parameters, insert the parameter-efficient structure adapter, and fine-tune it. Updating only 0.6\% of the overall parameters compared to full-model fine-tuning for each programming language, adapter tuning yields consistent improvements on code search and summarization tasks, achieving state-of-the-art results. In addition, we experimentally show its effectiveness in cross-lingual and low-resource scenarios. Multilingual fine-tuning with 200 samples per programming language approaches the results fine-tuned with the entire dataset on code summarization. Our experiments on three probing tasks show that adapter tuning significantly outperforms full-model fine-tuning and effectively overcomes catastrophic forgetting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑