arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.00399cs.SE
收藏

基于 Transformer 的软件工程技术实证研究

Empirical Study on Transformer-based Techniques for Software Engineering

Yan Xiao, Xinyue Zuo, Lei Xue, Kailong Wang, Jin Song Dong, Ivan Beschastnikh

AI总结:

本文实证比较 CodeBERT、CodeGPT 与 CodeT5 在四类软件工程任务上的适用性、泛化性和资源消耗,挑战编码器-解码器架构最优的固有认知。

摘要:

许多基于 Transformer 的代码预训练模型已经被开发出来,并应用于与代码相关的任务。本文回顾现有文献,考察模型架构对不同任务的适用性,并研究模型在不同数据集上的泛化能力及其资源消耗。我们考察了三个极具代表性的代码预训练模型:CodeBERT、CodeGPT 和 CodeT5,并在文献调研中发现的最受关注的前四项软件工程任务上开展实验:代码摘要、缺陷修复、缺陷检测和代码搜索。在研究中,我们在最先进评估指标下展示了仅解码器模型(CodeGPT)在特定生成任务上的能力,并质疑编码器-解码器架构最适合通用编码任务这一普遍看法。此外,我们发现最常使用的模型未必最适合某些应用,而且当前研究未能充分满足开发者需求。我们还发现,缺陷修复和代码摘要的基准数据集及常用数据集都无法使模型泛化到同一任务的其他数据集上(常用数据集指文献中除基准外使用频率最高的数据集)。我们使用统计检验来支持实验结论。最后,CodeBERT 在理解任务上非常高效,而 CodeT5 在生成任务上的效率令人怀疑,因为最高资源消耗并不能保证其在不同指标上持续取得更优性能。我们还讨论了未来推进基于 Transformer 的代码相关任务模型研究中的大量实际问题。

英文摘要:

Many Transformer-based pre-trained models for code have been developed and applied to code-related tasks. In this paper, we review the existing literature, examine the suitability of model architectures for different tasks, and look at the generalization ability of models on different datasets, and their resource consumption. We examine three very representative pre-trained models for code: CodeBERT, CodeGPT, and CodeT5, and conduct experiments on the top-4 most targeted software engineering tasks that we found in our literature survey: Code Summarization, Bug Fixing, Bug Detection, and Code Search. In our study, we showcase the capability of decoder-only models (CodeGPT) for specific generation tasks under state-of-the-art evaluation metrics and contest the common belief that the encoder-decoder architecture is optimal for general-purpose coding tasks. Additionally, we found that the most frequently used models are not necessarily the most suitable for certain applications and the developers' needs are not adequately addressed by current research. As well, we found that the benchmark and frequent dataset for Bug Fixing and Code Summarization both fail to enable models to generalize onto other datasets for the same task (the frequent dataset refers to the dataset with the highest frequency used in literature other than the benchmark). We use statistical testing to support our conclusions from experiments. Finally, CodeBERT is highly efficient for understanding tasks, whereas CodeT5's efficiency for generation tasks is in doubt, as the highest resource consumption does not guarantee a consistent better performance on different metrics. We also discuss the numerous practical issues in advancing future research on transformer-based models for code-related tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑