arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.13851cs.SEcs.AI
收藏

评估LLM生成的ACSL注解用于形式验证

Evaluating LLM-Generated ACSL Annotations for Formal Verification

Maynooth University(迈诺特大学)

Arshad Beg, Diarmuid O'Donoghue, Rosemary Monahan

AI总结:

本文评估了自动ACSL注解生成策略,比较了规则基Python脚本、Frama-C的RTE插件及三种LLM,发现规则方法更可靠,LLM性能波动大。

摘要:

形式规范对于构建可验证和可靠的软件系统至关重要,但为现实世界的C程序生成准确且可验证的规范仍具挑战性。本文呈现了一项经验性评估,比较了自动ACSL注解生成策略,包括规则基Python脚本、Frama-C的RTE插件以及三种大型语言模型(DeepSeek-V3.2、GPT-5.2和OLMo 3.1 32B Instruct)。研究聚焦于单次注解生成,评估这些方法在直接应用于验证任务时的表现。使用CASP基准的过滤子集,通过Frama-C的WP插件和多个SMT求解器评估生成注解,分析证明成功率、求解器超时和内部处理时间。结果表明,规则方法在验证成功方面仍更可靠,而LLM方法表现更不稳定。这些发现突显了LLM作为自动化规范生成互补工具的潜力和当前限制。

英文摘要:

Formal specifications are crucial for building verifiable and dependable software systems, yet generating accurate and verifiable specifications for real-world C programs remains challenging. This paper presents an empirical evaluation of automated ACSL annotation generation strategies for C programs, comparing a rule-based Python script, Frama-C's RTE plugin, and three large language models (DeepSeek-V3.2, GPT-5.2, and OLMo 3.1 32B Instruct). The study focuses on one-shot annotation generation, assessing how these approaches perform when directly applied to verification tasks. Using a filtered subset of the CASP benchmark, we evaluate generated annotations through Frama-C's WP plugin with multiple SMT solvers, analyzing proof success rates, solver timeouts, and internal processing time. Our results show that rule-based approaches remain more reliable for verification success, while LLM-based methods exhibit more variable performance. These findings highlight both the current limitations and the potential of LLMs as complementary tools for automated specification generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑