arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.16320cs.SEcs.AIcs.LG
收藏

大语言模型如何理解执行语义的鲁棒性?

How Robustly do LLMs Understand Execution Semantics?

University College London(伦敦大学学院)

Claudio Spiess, Prem Devanbu, Earl T. Barr

AI总结:

研究通过代码输出预测任务评估大语言模型对代码理解的鲁棒性,发现开源模型在代码变换和输入扰动下表现稳定,而前沿模型GPT-5.2表现出显著的脆弱性,且异常处理能力不足。

摘要:

大语言模型展现出卓越的推理能力,但其是依赖内部世界模型还是复杂的模式匹配仍存疑问。我们通过标准程序-输出预测任务研究大语言模型对代码理解的鲁棒性。结果显示,开源推理模型(DeepSeek-R1家族)在代码变换和输入扰动下保持稳定但准确率较低(38%至67%),而前沿模型GPT-5.2表现出显著的脆弱性。尽管在原始未扰动的CRUXEval基准上达到近完美的99%准确率,但扰动输入导致准确率下降20%至24%。此外,我们发现许多模型在预测引发异常的扰动输入时表现较差,且预测性能取决于异常类型。我们研究了缓解异常预测缺陷的对策,并评估了这些对策对预测非异常行为能力的影响。研究结果既指出了所有模型理解代码方式的局限性,也确立了使用扰动评估代码模型价值。

英文摘要:

LLMs demonstrate remarkable reasoning capabilities, yet whether they utilize internal world models or rely on sophisticated pattern matching remains open. We study LLMs through the lens of robustness of their code understanding using a standard program-output prediction task. Our results reveal a stark divergence in model behavior: while open-source reasoning models (DeepSeek-R1 family) maintain stable, albeit somewhat lower accuracies (38% to 67%) under code transformations & input perturbations, the frontier model GPT-5.2 exhibits significant brittleness. Despite achieving a near-perfect score of 99% on the original, unperturbed CRUXEval benchmark, perturbed inputs trigger accuracy declines between 20% and 24%. In addition, we find that many models perform much worse at predicting behavior on perturbed inputs that raise exceptions, and that prediction performance depends on the kind of exception. We study remedies to address this deficiency in exception prediction, and evaluate the effect of these remedies on the ability to predict non-exception behaviors. Our findings both point to limitations in the way all models understand code, and establish the value of using perturbation to evaluate code models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑