arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.07221cs.SE
收藏

基于生成的代码审查自动化:我们离成功还有多远?

Generation-based Code Review Automation: How Far Are We?

Xin Zhou, Kisub Kim, Bowen Xu, DongGyun Han, Junda He, David Lo

摘要:

代码评审是一种有效的软件质量保证活动,但它是劳动密集型和耗时的。因此,最近已经提出了许多基于生成的自动代码审查(ACR)方法,其利用深度学习技术来自动化代码审查过程中的各种活动(例如,代码修订生成和评审注释生成)。我们发现以前的作品有三个主要的局限性。首先,ACR方法已被证明是有益的,在每一个工作,但这些方法没有全面地相互比较,以显示其优越性比他们的同行ACR方法。其次,通用的预训练模型,如CodeT 5,被证明是有效的,在广泛的软件工程(SE)任务。然而,还没有先前的工作调查这些模型在ACR任务的有效性。第三,先前的工作严重依赖于精确匹配(EM)指标,该指标只关注完美的预测,而忽略了不完整答案所取得的积极进展。为了填补这一研究空白,我们通过比较最近的ACR工具以及通用预训练模型的有效性进行了全面的研究。结果表明,在大多数情况下,通用预训练模型CodeT 5的性能优于其他模型。具体来说,CodeT 5在两个代码修订生成任务中的性能比现有技术高13.4%-38.9%。此外,我们引入了一个新的度量标准,即编辑进度(EP),以量化ACR工具所取得的部分进展。结果表明,每个任务的模型的排名可以根据是否使用EM或EP而改变。最后,我们从实验结果中获得了一些有见地的经验教训,并揭示了基于生成的代码审查自动化的未来研究方向。

英文摘要:

Code review is an effective software quality assurance activity; however, it is labor-intensive and time-consuming. Thus, a number of generation-based automatic code review (ACR) approaches have been proposed recently, which leverage deep learning techniques to automate various activities in the code review process (e.g., code revision generation and review comment generation). We find the previous works carry three main limitations. First, the ACR approaches have been shown to be beneficial in each work, but those methods are not comprehensively compared with each other to show their superiority over their peer ACR approaches. Second, general-purpose pre-trained models such as CodeT5 are proven to be effective in a wide range of Software Engineering (SE) tasks. However, no prior work has investigated the effectiveness of these models in ACR tasks yet. Third, prior works heavily rely on the Exact Match (EM) metric which only focuses on the perfect predictions and ignores the positive progress made by incomplete answers. To fill such a research gap, we conduct a comprehensive study by comparing the effectiveness of recent ACR tools as well as the general-purpose pre-trained models. The results show that a general-purpose pre-trained model CodeT5 can outperform other models in most cases. Specifically, CodeT5 outperforms the prior state-of-the-art by 13.4\%--38.9\% in two code revision generation tasks. In addition, we introduce a new metric namely Edit Progress (EP) to quantify the partial progress made by ACR tools. The results show that the rankings of models for each task could be changed according to whether EM or EP is being utilized. Lastly, we derive several insightful lessons from the experimental results and reveal future research directions for generation-based code review automation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑