arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.04779cs.PLcs.AIcs.SE
收藏

LLM 能对程序语义进行推理吗?对 LLM 在形式化规约推断上的全面评估

Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference

School of Computing and Information Systems(计算与信息系统学院);The University of Melbourne(墨尔本大学)

Thanh Le-Cong, Bach Le, Toby Murray

AI总结:

本文引入综合基准 FormalBench 评估 LLM 对程序语义的推理能力,发现其在复杂结构(如循环)和语义保持变换上表现不佳,并设计自修复提示将成功率提升 25%。

摘要:

大型语言模型(LLM)正越来越多地被用于自动化编程任务。然而,LLM 在程序语义推理方面的能力仍未得到充分研究,留下了巨大的进一步探索空间。本文引入了 FormalBench,这是一个旨在评估 LLM 对程序语义推理能力的综合基准,特别是通过合成形式化程序规约以辅助验证程序正确性的任务。该任务既需要对所有可能的程序执行进行全面推理,又需要生成符合形式语法和语义的精确且语法正确的表达式。使用该基准,我们评估了 LLM 合成一致且完整规约的能力。我们的研究结果表明,LLM 在简单控制流上表现良好,但在更复杂的结构(尤其是循环)上表现挣扎,即使使用高级提示也是如此。此外,LLM 对语义保持变换表现出有限的鲁棒性。我们还强调了常见的失败模式,并设计了自修复提示,将成功率提高了 25%。

英文摘要:

Large Language Models (LLMs) are increasingly being used to automate programming tasks. Yet, LLMs' capabilities in reasoning about program semantics are still inadequately studied, leaving significant potential for further exploration. This paper introduces FormalBench, a comprehensive benchmark designed to evaluate LLMs' reasoning abilities on program semantics, particularly via the task of synthesizing formal program specifications to assist verifying program correctness. This task requires both comprehensive reasoning over all possible program executions and the generation of precise, syntactically correct expressions that adhere to formal syntax and semantics. Using this benchmark, we evaluated the ability of LLMs in synthesizing consistent and complete specifications. Our findings show that LLMs perform well with simple control flows but struggle with more complex structures, especially loops, even with advanced prompting. Additionally, LLMs exhibit limited robustness against semantic-preserving transformations. We also highlight common failure patterns and design self-repair prompts, improving success rates by 25%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑