arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13377cs.SE
收藏

预大语言模型和大语言模型时代代码审查基准和评估实践的综述

A Survey of Code Review Benchmarks and Evaluation Practices in Pre-LLM and LLM Era

University of Manitoba(曼尼托巴大学);Huawei Canada(华为加拿大);Concordia University(康考迪亚大学)

Taufiqul Islam Khan, Shaowei Wang, Haoxiang Zhang, Tse-Hsun Chen

AI总结:

本文综述了预LLM和LLM时代代码审查基准和评估实践,提出多级分类法,揭示了审查任务的转变及未来研究方向。

摘要:

代码审查是现代软件工程中的关键实践,帮助开发人员在早期发现缺陷,提高代码质量并促进知识共享。随着大型语言模型(LLMs)的快速发展,越来越多的工作探索了自动化支持代码审查。然而,该领域的进展受到缺乏系统理解现有基准和评估实践的阻碍。当前的代码审查数据集分散,设计差异大,提供的洞察有限,无法明确评估实际的审查能力。在本文中,我们全面回顾了涵盖预LLM和LLM时代的代码审查基准(2015-2025)。我们分析了99篇研究论文(58篇预LLM时代和41篇LLM时代)并提取了关键元数据,包括数据集、评估指标、数据来源和目标任务。基于此分析,我们提出一个多级分类法,将代码审查研究分为五个领域和18个细粒度任务。我们的研究揭示了向端到端生成同行审查的明显转变,增加了多语言覆盖,并减少了独立变更理解任务。我们进一步识别了当前基准的局限性,并概述了未来方向,包括更广泛的任务覆盖、动态运行时评估和分类法引导的细粒度评估。本文的综述为开发更现实和全面的LLM代码审查基准提供了结构化基础。

英文摘要:

Code review is a critical practice in modern software engineering, helping developers detect defects early, improve code quality, and facilitate knowledge sharing. With the rapid advancement of large language models (LLMs), a growing body of work has explored automated support for code review. However, progress in this area is hindered by the lack of a systematic understanding of existing benchmarks and evaluation practices. Current code review datasets are scattered, vary widely in design, and provide limited insight into what review capabilities are actually being assessed. In this paper, we present a comprehensive survey of code review benchmarks spanning both the Pre-LLM and LLM eras (2015--2025). We analyze 99 research papers (58 Pre-LLM era and 41 LLM era) and extract key metadata, including datasets, evaluation metrics, data sources, and target tasks. Based on this analysis, we propose a multi-level taxonomy that organizes code review research into five domains and 18 fine-grained tasks. Our study reveals a clear shift toward end-to-end generative peer review, increasing multilingual coverage, and a decline in standalone change understanding tasks. We further identify limitations of current benchmarks and outline future directions, including broader task coverage, dynamic runtime evaluation, and taxonomy-guided fine-grained assessment. This survey provides a structured foundation for developing more realistic and comprehensive benchmarks for LLM-based code review.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑