精炼LLM代码生成中的批判性思维：一个基于错误前提的评估框架
Refining Critical Thinking in LLM Code Generation: A Faulty Premise-based Evaluation Framework
AI总结：
本文提出FPBench框架，系统评估15个LLM在错误前提下的代码生成能力，发现模型自我审视不足、资源投入收益递减及认知机制三重分离，为可靠代码生成提供理论基础。
摘要：
随着大型语言模型（LLMs）代码生成能力的进步，其对输入前提的依赖程度显著增强。当用户提供包含错误前提的输入时，代码生成幻觉的发生概率大幅上升，暴露出模型在自我审视能力方面的不足。本文提出了错误前提基准（FPBench），这是首个针对错误前提的代码生成评估框架。通过系统构建三类错误前提并整合多维评估指标，该框架对15个具有代表性的LLM进行了深入评估。主要发现如下：（1）大多数模型在错误前提条件下表现出较差的推理能力和次优的代码生成性能，严重依赖显式提示进行错误检测，自我审视能力有限；（2）错误前提触发了资源投入的收益递减点，盲目增加生成长度无法提升质量；（3）三类错误前提分别激活了模型不同的缺陷模式，揭示了代码生成模型认知机制中的三重分离现象。本研究不仅凸显了LLM在代码生成中主动验证前提的迫切需求，而且通过所提出的FPBench框架和多维评估体系，为开发可靠、以人为中心的代码生成模型提供了理论基础和实践路径。
英文摘要：
With the advancement of code generation capabilities in large language models (LLMs), their reliance on input premises has intensified. When users provide inputs containing faulty premises, the probability of code generation hallucinations rises significantly, exposing deficiencies in their self-scrutiny capabilities. This paper proposes Faulty Premises Bench (FPBench), the first code generation evaluation framework targeting faulty premises. By systematically constructing three categories of faulty premises and integrating multi-dimensional evaluation metrics, it conducts in-depth assessments of 15 representative LLMs. The key findings are as follows: (1) Most models exhibit poor reasoning abilities and suboptimal code generation performance under faulty premises, heavily relying on explicit prompts for error detection, with limited self-scrutiny capabilities; (2) Faulty premises trigger a point of diminishing returns in resource investment, leading to blindly increasing length fails to enhance quality; (3) The three types of faulty premises respectively activate distinct defect patterns in models, revealing a triple dissociation in the cognitive mechanisms of code generation models. This study not only highlights the urgent need for LLMs to proactively verify premises in code generation but also, through the proposed FPBench framework and multi-dimensional evaluation system, provides a theoretical foundation and practical pathway for developing reliable, human-centric code generation models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
