LLMs是否可靠代码审查员？需求符合性判断中的系统性过度纠正
Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement
AI总结：
本文研究了LLMs在需求符合性判断中的可靠性问题，发现其存在系统性过度纠正现象，并提出Fix-guided Verification Filter以提高代码审查的可靠性。
摘要：
大型语言模型（LLMs）已成为软件开发中不可或缺的工具，被广泛用于需求工程、代码生成和审查任务。软件工程师经常依赖LLMs来验证代码实现是否满足任务要求，从而确保代码的健壮性和准确性。然而，仍然不清楚LLMs能否可靠地确定代码是否符合给定的任务描述，通常以自然语言规范的形式存在。在本文中，我们揭示了LLMs在匹配代码与自然语言要求时的系统性失败。具体而言，通过广泛采用的基准和统一提示设计，我们证明LLMs经常将正确的代码实现误判为不合规或缺陷。令人惊讶的是，我们发现更详细的提示设计，特别是那些要求解释和提出修正的提示，会导致更高的误判率，突显了基于LLM的代码助手的关键可靠性问题。我们进一步分析了这些失败的驱动机制，并评估了需要理由的判断的可靠性。基于这些发现，我们提出了一个Fix-guided Verification Filter，将模型提出的修复视为可执行的反事实证据，并使用基准测试和规范约束的增强测试验证原始和修订的实现。我们的结果揭示了之前未被充分探讨的LLM在代码审查能力中的限制，并为将基于LLM的审查员与在自动化审查和开发流水线中的保护措施相结合提供了实用指导。
英文摘要：
Large language models (LLMs) have become essential tools in software development, widely used for requirements engineering, code generation and review tasks. Software engineers often rely on LLMs to verify if code implementation satisfy task requirements, thereby ensuring code robustness and accuracy. However, it remains unclear whether LLMs can reliably determine code against the given task descriptions, which is usually in a form of natural language specifications. In this paper, we uncover a systematic failure of LLMs in matching code to natural language requirements. Specifically, with widely adopted benchmarks and unified prompts design, we demonstrate that LLMs frequently misclassify correct code implementation as non-compliant or defective. Surprisingly, we find that more detailed prompt design, particularly with those requiring explanations and proposed corrections, leads to higher misjudgment rates, highlighting critical reliability issues for LLM-based code assistants. We further analyze the mechanisms driving these failures and evaluate the reliability of rationale-required judgments. Building on these findings, we propose a Fix-guided Verification Filter that treats the model proposed fix as executable counterfactual evidence, and validates the original and revised implementations using benchmark tests and spec-constrained augmented tests. Our results expose previously under-explored limitations in LLM-based code review capabilities, and provide practical guidance for integrating LLM-based reviewers with safeguards in automated review and development pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
