arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.16792cs.LOcs.AI
收藏

Laurel:利用大型语言模型解除自动化验证的阻塞

Laurel: Unblocking Automated Verification with Large Language Models

UC San Diego(加州大学圣迭戈分校)

Eric Mugnier, Emmanuel Anaya Gonzalez, Ranjit Jhala, Nadia Polikarpova, Yuanyuan Zhou

AI总结:

本文提出 Laurel 工具,通过设计两种特定领域的提示技术,利用大型语言模型自动生成断言,以解除 Dafny 等程序验证器在自动化证明中因缺乏提示而导致的阻塞,无需人工干预。

摘要:

诸如 Dafny 之类的程序验证器通过将证明外包给 SMT 求解器来实现证明的自动化。然而,这种自动化并不完美,求解器通常需要以断言形式提供的提示,这给证明工程师造成了负担。在本文中,我们提出了 Laurel,这是一种通过使用大型语言模型(LLMs)自动生成断言来减轻此负担的工具。为了提高 LLMs 在此任务中的成功率,我们设计了两种特定领域的提示技术。首先,我们通过分析验证器的错误消息并在该位置插入断言占位符,来帮助 LLM 确定缺失断言的位置。其次,我们向 LLM 提供来自同一代码库的示例断言,这些断言是基于一种新的证明相似度度量来选择的。我们在我们的新基准 DafnyGym 上评估了我们的技术,该基准是我们从三个真实世界的 Dafny 代码库中提取的复杂引理数据集。我们的评估表明,Laurel 仅需几次尝试就能生成超过 56.6% 的所需断言,使 LLMs 成为无需人工干预即可解除程序验证器阻塞的经济实惠的工具。

英文摘要:

Program verifiers such as Dafny automate proofs by outsourcing them to an SMT solver. This automation is not perfect, however, and the solver often requires hints in the form of assertions, creating a burden for the proof engineer. In this paper, we propose Laurel, a tool that alleviates this burden by automatically generating assertions using large language models (LLMs). To improve the success rate of LLMs in this task, we design two domain-specific prompting techniques. First, we help the LLM determine the location of the missing assertion by analyzing the verifier's error message and inserting an assertion placeholder at that location. Second, we provide the LLM with example assertions from the same codebase, which we select based on a new proof similarity metric. We evaluate our techniques on our new benchmark DafnyGym, a dataset of complex lemmas we extracted from three real-world Dafny codebases. Our evaluation shows that Laurel is able to generate over 56.6\% of the required assertions given only a few attempts, making LLMs an affordable tool for unblocking program verifiers without human intervention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑