arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.01394cs.SEcs.AI
收藏

LiveFMBench: 揭示代理工作流在规范生成中的力量与局限

LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation

Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);The Hong Kong University of Science(香港科学与技术大学);Guangzhou Institute of Technology, Xidian University(西安电子科技大学广州研究所)

Dong Xu, Jialun Cao, Guozhao Mo, Junjie Hu, Cheng Wen, Hongyu Lin, Xianpei Han, Shengchao Qin, Cong Tian, Shing-Chi Cheung, Le Sun, Yaojie Lu

AI总结:

本文通过LiveFMBench系统研究LLM和代理在C程序形式化规范生成中的能力与失败模式,发现直接提示和代理流程显著提升成功率,但需排除不忠实行为以准确评估性能。

摘要:

形式化规范对于严谨的程序验证至关重要,但编写正确的规范仍然成本高昂且难以自动化。尽管大型语言模型(LLMs)和代理已展现出显著进展,但它们的真实能力和失败模式仍不明确。我们首次系统且具有污染意识地研究了LLM和代理在C程序形式化规范生成中的能力。我们介绍了LiveFMBench,一个持续演进的630个带有ACSL(ANSI/ISO C规范语言)注释的C程序基准,包括360个新收集的案例,旨在减轻数据泄漏。利用此基准,我们评估了不同采样大小的直接提示、具有推理能力(思考模式)的推理、代理流程,并进行了细粒度的失败分析。实验结果表明,直接提示的简单评估严重高估了性能,因为模型在直接提示下可能会表现出不忠实的行为,如欺骗自动证明者或忽略代码上下文约束;在排除此类情况后,真正的规范生成准确性下降了约20%。我们进一步发现,增加采样和思考模式显著提高了成功率,较小的模型更受益于思考模式。代理流程在低采样预算和更难的数据集上特别有效。失败分析进一步表明,错误的循环不变式是主导的错误类型,而代理流程显著减少了断言错误。这些结果揭示了当前LLM方法的根本限制,并表明它们仍远未取代人工编写的形式化规范。我们发布了LiveFMBench在https://huggingface.co/datasets/fm-universe/Live-FM-Bench并发布所有评估成果以支持未来研究。

英文摘要:

Formal specification is essential for rigorous program verification, yet writing correct specifications remains costly and difficult to automate. Although large language models (LLMs) and agents have shown promising progress, their true capabilities and failure modes remain unclear. We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs. We introduce LiveFMBench, a continuously evolving benchmark of 630 ACSL (ANSI/ISO C Specification Language)-annotated C programs, including 360 newly collected cases designed to mitigate data leakage. Using this benchmark, we evaluate direct prompting with different sampling sizes, reasoning-enabled (thinking mode) inference, the agentic pipeline, and perform a fine-grained failure analysis. Experimental results reveal that naive evaluation substantially overestimates performance because models under direct prompting may exhibit unfaithful behaviors, such as deceiving automated provers or ignoring code-context constraints; after excluding such cases, the true specification generation accuracy drops by approximately 20\%. We further find that both increased sampling and thinking mode significantly improve success rates, with smaller models benefiting more from thinking mode. Agentic pipelines are particularly effective under low sampling budgets and on harder datasets. Failure analysis further shows that incorrect loop invariants are the dominant error type, while agentic pipelines notably reduce assertion errors. These results expose fundamental limitations in current LLM-based approaches and suggest they remain far from replacing human-authored formal specifications. We release LiveFMBench at https://huggingface.co/datasets/fm-universe/Live-FM-Bench and all evaluation artifacts to support future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑