arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.12079cs.SE
收藏

方向性扩散-风格代码编辑预训练

Directional Diffusion-Style Code Editing Pre-training

Key Lab of HCST (PKU), MOE(智能技术与系统国家重点实验室(北京大学),教育部;计算机科学与技术系,北京大学);SCS, Peking University

Qingyuan Liang, Zeyu Sun, Qihao Zhu, Junhao Hu, Yifan Zhao, Yizhou Chen, Mingxuan Zhu, Guoqing Wang, Lu Zhang

AI总结:

DivoT5通过方向扩散方法提升代码编辑预训练效果,实现多个任务的SOTA性能。

摘要:

代码预训练模型在各种软件工程任务中已展现出有前景的有效性。在这些任务中,许多任务与软件进化和/或代码编辑有关。然而,现有的代码预训练模型往往忽略了现实中的代码编辑数据以及编辑过程的进化性质。在本文中,为了模拟人类开发人员的逐步代码编辑过程,我们提出了DivoT5,一种基于数据层面方向扩散的预训练模型。在DivoT5中,我们采用了两类预训练任务。第一类是带有扩散方向表示代码进化的掩码和去噪任务。也就是说,我们首先对代码片段应用一个噪声过程,然后再让预训练过程将带有噪声的片段恢复成经过进化后的代码片段。第二类是旨在强化进化方向的任务。也就是说,我们首先为每对前后进化片段生成各种中间版本,然后让预训练过程将这些中间版本转换成每对的片段后版本。我们使用五个下游任务对DivoT5进行了两种代码编辑场景和一个非编辑场景的评估。给定每个下游任务,我们对预训练的DivoT5进行微调以评估其有效性。我们的实验结果表明,DivoT5在大多数任务上相对于同规模(220M)、大尺寸(770M)模型在微调中,以及在少样本设置中相对于十亿级(6.7B、8B、ChatGPT)模型均取得了最先进(SOTA)的表现。对于一个代码编辑任务(即自动代码审查),DivoT5在CodeT5-small(60M)上预训练后,甚至能超越CodeT5-base(220M)和其他具有220M参数的预训练模型,除了DivoT5在CodeT5-base(220M)上预训练的情况。

英文摘要:

Code pre-trained models have shown promising effectiveness in various software engineering tasks. Among these tasks, many tasks are related to software evolution and/or code editing. However, existing code pre-trained models often overlook the real-world code editing data and the evolutionary nature of the editing process. In this paper, to simulate the step-by-step code editing process of human developers, we propose DivoT5, a pre-trained model based on directional diffusion at the data level. In DivoT5, we adopt two categories of pre-training tasks. The first category is mask and denoising tasks augmented with a diffusion direction representing code evolution. That is, we first apply a noising process to the code snippets before evolution, and then ask the pre-training process to restore the snippets with noise into the code snippets after evolution. The second category is tasks aiming to reinforce the evolutionary direction. That is, we first generate various intermediate versions for each pair of snippets before and after evolution, and then ask the pre-training process to transform the intermediate versions into the snippet after evolution for each pair. We evaluate DivoT5 for two code-editing scenarios and one non-editing scenario using five downstream tasks. Given each downstream task, we fine-tune the pre-trained DivoT5 to evaluate its effectiveness. Our experimental results show that DivoT5 achieves state-of-the-art (SOTA) performance on most tasks in comparison to models of the same scale (220M), large scale (770M) models in fine-tuning, and billion-scale (6.7B, 8B, ChatGPT) models in few-shot settings. For one code-editing task (i.e., automated code review), DivoT5 pre-trained on top of CodeT5-small (60M) can even outperform CodeT5-base (220M) and other pre-trained models with 220M parameters except for DivoT5 pre-trained on top of CodeT5-base (220M).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑