CodeCrash：揭示LLM在代码推理中对误导性自然语言的脆弱性
CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
AI总结：
提出压力测试框架CodeCrash，评估17个LLM在结构扰动和误导性自然语言下的代码推理鲁棒性，发现模型过度依赖自然语言线索导致性能显著下降，并揭示了大型推理模型在错误提示下因病态自我反思引发的推理崩溃现象。
摘要：
大型语言模型（LLMs）近期在代码相关任务中展现出强大能力，但其在扰动下代码推理的鲁棒性仍有待深入探索。我们引入了CodeCrash，这是一个包含来自CruxEval和LiveCodeBench的1,279个问题的压力测试框架，旨在评估模型在结构扰动和误导性自然语言（NL）上下文下的推理可靠性。通过对17个LLMs的系统评估，我们发现模型经常过度依赖NL线索作为推理捷径，导致在输出预测任务中平均性能下降23.2%。即使采用思维链推理，由于易分心和合理化倾向，模型平均仍出现13.8%的性能下降，这表明其缺乏区分实际代码行为的关键推理能力。尽管具备内部推理机制的大型推理模型通过培养批判性思维提升了鲁棒性，但看似合理却错误的提示会引发病态的自我反思，导致令牌消耗增加2-3倍，在极端情况下甚至会使QwQ-32B出现灾难性的认知失调。我们将此现象称为推理崩溃。CodeCrash为评估代码推理的鲁棒性提供了严格的基准，指导未来的研究与开发朝着更可靠、更具弹性的模型迈进。
英文摘要：
Large Language Models (LLMs) have recently demonstrated strong capabilities in code-related tasks, but their robustness in code reasoning under perturbations remains underexplored. We introduce CodeCrash, a stress-testing framework with 1,279 questions from CruxEval and LiveCodeBench, designed to evaluate reasoning reliability under structural perturbations and misleading natural language (NL) contexts. Through a systematic evaluation of 17 LLMs, we find that models often shortcut reasoning by over-relying on NL cues, leading to an average performance degradation of 23.2% in output prediction tasks. Even with Chain-of-Thought reasoning, models on average still have a 13.8% drop due to distractibility and rationalization, revealing a lack of critical reasoning capability to distinguish the actual code behaviors. While Large Reasoning Models with internal reasoning mechanisms improve robustness by fostering critical thinking, plausible yet incorrect hints can trigger pathological self-reflection, causing 2-3 times token consumption and even catastrophic cognitive dissonance in extreme cases for QwQ-32B. We refer to this phenomenon as Reasoning Collapse. CodeCrash provides a rigorous benchmark for evaluating robustness in code reasoning, guiding future research and development toward more reliable and resilient models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
