基于实践丰富上下文的LLM细粒度代码审查基准测试
Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice
AI总结：
ContextCRBench通过丰富上下文的细粒度代码审查基准测试，评估LLM在代码审查中的性能，发现文本上下文比代码上下文更有效，且在工业应用中提升了审查系统性能。
摘要：
代码审查是软件质量保证的基石，近年来大型语言模型（LLM）在自动化方面的进展显示出潜力。然而，现有的基于LLM的代码审查基准测试面临三大主要限制。语义上下文不足：大多数基准仅提供代码差异，而不包含诸如问题描述等文本信息，这些信息对于理解开发者意图至关重要。数据质量问题是：没有严格验证，许多样本存在噪声——例如，对过时或不相关代码的审查，减少了评估的可靠性。粗粒度：大多数基准在文件或提交级别操作，忽略了细粒度、行级推理对于精确审查的重要性。我们引入ContextCRBench，一个高质量、上下文丰富的基准，用于细粒度LLM在代码审查中的评估。我们的构建流程包括：原始数据爬取，从顶级仓库收集153.7万个问题和拉取请求；全面上下文提取，链接问题-PR对以获取文本上下文，并提取完整的周围函数或类以获取代码上下文；以及多阶段数据过滤，结合规则基础和LLM基础验证以删除过时、格式错误或低价值的样本，最终得到67,910个上下文丰富的条目。ContextCRBench支持三种评估场景，与审查工作流程对齐：补丁级别质量评估、行级缺陷定位以及行级评论生成。评估八种领先的LLM（四款封闭源和四款开源）发现，文本上下文比单独的代码上下文产生更大的性能提升，而当前LLM仍远未达到人类水平的审查能力。在字节跳动部署后，ContextCRBench推动了一个自我演进的代码审查系统，性能提升了61.98%，并展示了其鲁棒性和工业实用性。
英文摘要：
Code review is a cornerstone of software quality assurance, and recent advances in Large Language Models (LLMs) have shown promise in its automation. However, existing benchmarks for LLM-based code review face three major limitations. Lack of semantic context: most benchmarks provide only code diffs without textual information such as issue descriptions, which are crucial for understanding developer intent. Data quality issues: without rigorous validation, many samples are noisy-e.g., reviews on outdated or irrelevant code-reducing evaluation reliability. Coarse granularity: most benchmarks operate at the file or commit level, overlooking the fine-grained, line-level reasoning essential for precise review. We introduce ContextCRBench, a high-quality, context-rich benchmark for fine-grained LLM evaluation in code review. Our construction pipeline comprises: Raw Data Crawling, collecting 153.7K issues and pull requests from top-tier repositories; Comprehensive Context Extraction, linking issue-PR pairs for textual context and extracting the full surrounding function or class for code context; and Multi-stage Data Filtering, combining rule-based and LLM-based validation to remove outdated, malformed, or low-value samples, resulting in 67,910 context-enriched entries. ContextCRBench supports three evaluation scenarios aligned with the review workflow: hunk-level quality assessment, line-level defect localization, and line-level comment generation. Evaluating eight leading LLMs (four closed-source and four open-source) reveals that textual context yields greater performance gains than code context alone, while current LLMs remain far from human-level review ability. Deployed at ByteDance, ContextCRBench drives a self-evolving code review system, improving performance by 61.98% and demonstrating its robustness and industrial utility. https://github.com/kinesiatricssxilm14/ContextCRBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
