arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.00280cs.SEcs.AI
收藏

VeriAct:超越可验证性——基于代理的正确且完整的正式规范合成

Spec-Harness: Assessing and Guiding LLM-Synthesized Formal Specifications Beyond Verifiability

University of California, Irvine(加利福尼亚大学尔湾分校)

Md Rakib Hossain Misu, Cristina V. Lopes, Iris Ma, Yuangang Li, Shuvendu Lahiri

AI总结:

本文提出VeriAct框架,通过迭代合成与修复规范,超越传统方法的性能极限,生成正确且完整的正式规范。

摘要:

正式规范在确保软件可靠性和正确性方面起着核心作用。然而,自动合成高质量正式规范仍是一个具有挑战性的任务,通常需要领域专业知识。最近的工作已应用大语言模型生成Java建模语言(JML)的规范,报告了较高的验证通过率。但通过验证器是否意味着规范实际上正确且完整?在本工作中,我们首先对经典和基于提示的方法进行综合评估,比较自动JML规范合成。然后研究提示优化是否能进一步提升合成质量,通过结构化验证反馈进化提示。尽管优化提高了验证器通过率,但我们发现存在明显的性能上限。更关键的是,我们提出了Spec-Harness评估框架,通过符号验证测量规范的正确性和完整性,揭示大量通过验证器的规范,包括优化后的,实际上是错误或不完整的,以不可见的方式过度或不足地约束输入和输出。为了超越这一上限,我们提出VeriAct,一种由验证引导的代理框架,通过LLM驱动的规划、代码执行、验证和Spec-Harness反馈的闭环迭代合成和修复规范。在两个基准数据集上的实验表明,VeriAct优于基于提示和优化的基线,生成的规范不仅可验证,而且正确且完整。

英文摘要:

Formal specifications play a central role in ensuring software reliability, yet automatically synthesizing high-quality specifications remains difficult and often requires domain expertise. Recent work has applied large language models to generate specifications in the Java Modeling Language (JML), reporting high verifier pass rates. But passing a verifier only confirms that an implementation is consistent with a specification, not that the specification is meaningful: a trivial postcondition such as ensures true satisfies any verifier while saying nothing about the code. Mutation-based completeness scores go further, but they return a single number and cannot tell a generator which clause to fix. In this work, we first compare classical and prompt-based JML synthesis under a unified setup, and find that 22.7% of the verified specifications have no postcondition at all, and that prompt optimization through verification feedback repairs weak prompts but does not go beyond the best hand-written one. We then introduce Spec-Harness, a framework that measures the behavioral adequacy of a specification along four dimensions of precondition and postcondition correctness and completeness, using Hoare-triple based symbolic verification and input/output mutation on systematically built test suites. Spec-Harness reveals that many verifier-accepted specifications are behaviorally weak, over- or under-constraining inputs and outputs in ways the verifier cannot see. Finally, we show that Spec-Harness works as a feedback signal that helps coding agents synthesize specifications with higher behavioral adequacy on held-out tests. This holds for the general-purpose agents Codex and Claude Code and for a CodeAct-style JML synthesis agent that we built to use the verifier and Spec-Harness as feedback.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑