CRScore:将代码审查评论的自动化评估建立在代码声明与坏味基础上
CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells
AI总结:
本文提出无参考指标 CRScore,利用 LLM 与静态分析器识别的代码声明和潜在问题,从多维度细粒度评估代码审查评论,并发布 2.9k 条人工标注语料。
摘要:
自动化代码审查任务最近受到机器学习领域的广泛关注。然而,当前的审查评论评估指标依赖于将给定代码变更(也称为 diff)与人工撰写的参考内容进行比较。此外,代码审查与生成和摘要一样,是一个一对多问题:一个 diff 可能存在许多“有效审查”。因此,我们开发了 CRScore——一种无参考指标,用于衡量简洁性、全面性和相关性等审查质量维度。我们将 CRScore 设计为以 LLM 和静态分析器在代码中检测到的声明及潜在问题为依据来评估审查。我们证明,CRScore 能够生成有效、细粒度的审查质量评分;在开源指标中,这些评分与人类判断的一致性最高(Spearman 相关系数为 0.54),并且比基于参考的指标更敏感。我们还发布了一个包含 2.9k 条人工标注审查质量评分的语料库,涵盖机器生成和 GitHub 审查评论,以支持自动化指标的开发。
英文摘要:
The task of automated code review has recently gained a lot of attention from the machine learning community. However, current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). Furthermore, code review is a one-to-many problem, like generation and summarization, with many "valid reviews" for a diff. Thus, we develop CRScore - a reference-free metric to measure dimensions of review quality like conciseness, comprehensiveness, and relevance. We design CRScore to evaluate reviews in a way that is grounded in claims and potential issues detected in the code by LLMs and static analyzers. We demonstrate that CRScore can produce valid, fine-grained scores of review quality that have the greatest alignment with human judgment among open source metrics (0.54 Spearman correlation) and are more sensitive than reference-based metrics. We also release a corpus of 2.9k human-annotated review quality scores for machine-generated and GitHub review comments to support the development of automated metrics.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。