arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.19801cs.SEcs.AIcs.CL
收藏

CRScore:将代码审查评论的自动化评估建立在代码声明与坏味基础上

CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells

Carnegie Mellon University(卡内基梅隆大学);Language Technologies Institute(语言技术研究所)

Atharva Naik, Marcus Alenius, Daniel Fried, Carolyn Rose

AI总结:

本文提出无参考指标 CRScore,利用 LLM 与静态分析器识别的代码声明和潜在问题,从多维度细粒度评估代码审查评论,并发布 2.9k 条人工标注语料。

摘要:

自动化代码审查任务最近受到机器学习领域的广泛关注。然而,当前的审查评论评估指标依赖于将给定代码变更(也称为 diff)与人工撰写的参考内容进行比较。此外,代码审查与生成和摘要一样,是一个一对多问题:一个 diff 可能存在许多“有效审查”。因此,我们开发了 CRScore——一种无参考指标,用于衡量简洁性、全面性和相关性等审查质量维度。我们将 CRScore 设计为以 LLM 和静态分析器在代码中检测到的声明及潜在问题为依据来评估审查。我们证明,CRScore 能够生成有效、细粒度的审查质量评分;在开源指标中,这些评分与人类判断的一致性最高(Spearman 相关系数为 0.54),并且比基于参考的指标更敏感。我们还发布了一个包含 2.9k 条人工标注审查质量评分的语料库,涵盖机器生成和 GitHub 审查评论,以支持自动化指标的开发。

英文摘要:

The task of automated code review has recently gained a lot of attention from the machine learning community. However, current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). Furthermore, code review is a one-to-many problem, like generation and summarization, with many "valid reviews" for a diff. Thus, we develop CRScore - a reference-free metric to measure dimensions of review quality like conciseness, comprehensiveness, and relevance. We design CRScore to evaluate reviews in a way that is grounded in claims and potential issues detected in the code by LLMs and static analyzers. We demonstrate that CRScore can produce valid, fine-grained scores of review quality that have the greatest alignment with human judgment among open source metrics (0.54 Spearman correlation) and are more sensitive than reference-based metrics. We also release a corpus of 2.9k human-annotated review quality scores for machine-generated and GitHub review comments to support the development of automated metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑