代码评审生成方法的深度评估:超越词汇相似性
Deep Assessment of Code Review Generation Approaches: Beyond Lexical Similarity
AI总结:
针对代码评审自动评估过度依赖词汇相似性的问题,本文构建 GradedReviews 基准,并提出向量语义相似性与 ChatGPT 评分两种方法,将与人评的相关系数从0.22提升至0.47。
摘要:
代码评审是确保软件项目质量的标准实践,近期研究广泛关注自动化代码评审。尽管在生成代码评审方面已取得显著进展,但对这些评审的自动化评估仍探索不足,现有方法和指标往往被证明不够准确。当前指标(如 BLEU)主要依赖生成评审与参考评审之间的词汇相似性。然而,此类指标往往会低估那些以不同于参考评审的方式阐明预期问题的评审。本文探讨生成评审与参考评审之间的语义相似性如何增强代码评审的自动化评估。我们首先提出一个名为 GradedReviews 的基准,其构建方式是收集开源项目中的真实代码评审、使用最先进方法生成评审,并人工评估其质量。随后,我们使用该基准评估现有的代码评审评估指标,揭示其局限性。为解决这些局限,我们提出两种新颖的基于语义的代码评审评估方法。第一种方法使用深度学习模型将生成评审及其参考评审转换为数字向量,然后通过余弦相似性度量其语义相似性。第二种方法根据生成评审及其参考评审生成提示,将该提示提交给 ChatGPT,并要求 ChatGPT 按照明确定义的标准对生成评审进行评分。我们在 GradedReviews 基准上的评估表明,所提出的基于语义的方法在评估生成的代码评审方面显著优于现有最先进指标,将所得分数与人工分数之间的相关系数从 0.22 提升至 0.47。
英文摘要:
Code review is a standard practice for ensuring the quality of software projects, and recent research has focused extensively on automated code review. While significant advancements have been made in generating code reviews, the automated assessment of these reviews remains less explored, with existing approaches and metrics often proving inaccurate. Current metrics, such as BLEU, primarily rely on lexical similarity between generated and reference reviews. However, such metrics tend to underestimate reviews that articulate the expected issues in ways different from the references. In this paper, we explore how semantic similarity between generated and reference reviews can enhance the automated assessment of code reviews. We first present a benchmark called \textit{GradedReviews}, which is constructed by collecting real-world code reviews from open-source projects, generating reviews using state-of-the-art approaches, and manually assessing their quality. We then evaluate existing metrics for code review assessment using this benchmark, revealing their limitations. To address these limitations, we propose two novel semantic-based approaches for assessing code reviews. The first approach involves converting both the generated review and its reference into digital vectors using a deep learning model and then measuring their semantic similarity through Cosine similarity. The second approach generates a prompt based on the generated review and its reference, submits this prompt to ChatGPT, and requests ChatGPT to rate the generated review according to explicitly defined criteria. Our evaluation on the \textit{GradedReviews} benchmark indicates that the proposed semantic-based approaches significantly outperform existing state-of-the-art metrics in assessing generated code review, improving the correlation coefficient between the resulting scores and human scores from 0.22 to 0.47.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。