发表机构
Sun Yat-sen University; Chongqing University; Huawei Cloud Computing Technologies Co., Ltd.（中山大学; 重庆大学; 华为云计算技术有限公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究推出首个面向真实多轮代码审查的缺陷状态感知基准MCR-Bench，通过实验发现主流LLMs在该任务中存在整体能力有限、缺陷敏感性性能差异大及存在跨轮次时间错位等失效机制的问题。
AI 中文摘要
在真实世界软件开发中，代码审查通常涉及开发者与审查者之间的迭代交互，以提升软件质量，这使得该过程成本高昂且耗时。尽管近期研究探索了大型语言模型（LLMs）用于自动化代码审查，但大多数方法将代码审查过度简化为单轮静态决策任务，未能捕捉到真实审查场景中固有的多轮交互性质与复杂问题解决过程。为弥合这一差距，我们推出MCR-Bench，这是首个面向真实多轮代码审查的缺陷状态感知基准。MCR-Bench涵盖五种常用编程语言，包含2269个真实世界多轮代码审查任务，每个任务均标注了细粒度缺陷信息与跨轮次状态标签。MCR-Bench中的每个任务都配备了细粒度缺陷元数据（如描述、类型、严重程度）以及动态状态标注，捕捉缺陷在多轮过程中的完整演化轨迹。我们通过在主流LLMs上对MCR-Bench进行大量实验，得出若干发现：（1）整体能力有限：实验显示，主流LLMs在缺陷检测与缺陷生命周期状态跟踪方面的整体性能有限，且性能随交互轮数增加显著下降；（2）缺陷敏感性性能：LLMs在不同缺陷类型与严重程度上的性能差异显著，语义复杂或低显著性缺陷更易被遗漏；（3）潜在失效机制：我们的深入错误分析剖析了假阳性与假阴性的不同驱动因素，揭示了跨轮次时间错位、长程记忆不足等关键弱点。
英文摘要
In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios. To bridge this gap, we introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming languages and consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information and cross-round state labels. Each task in MCR-Bench is equipped with fine-grained defect metadata (e.g., description, type, severity) alongside dynamic state annotations, capturing the complete evolutionary trajectory of a defect throughout the multi-round process. We obtain several findings through extensive experiments on MCR-Bench with mainstream LLMs. (1) Limited overall capability: experiments reveal that mainstream LLMs exhibit limited overall performance in defect detection and defect lifecycle state tracking, with performance degrading significantly as the number of interaction rounds increases; (2) Defect-sensitive performance: LLMs' performance varies substantially across different defect types and severity levels, with semantically complex or low-salience defects being significantly more likely to be missed; (3) Underlying Failure Mechanisms: our in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.
CommentsAccepted at ISSTA 2026
