迈向 AI 辅助合成经过验证的 Dafny 方法
Towards AI-Assisted Synthesis of Verified Dafny Methods
AI总结:
本文通过无上下文、签名和检索增强思维链三种提示方式,提升 GPT-4 与 PaLM-2 在 Dafny 验证式编程上的能力,并为 MBPP 问题贡献了 153 个经过验证的 Dafny 解答。
摘要:
大型语言模型在包括编程在内的许多领域展现出巨大前景。做出承诺容易,但兑现承诺很难,语言模型常常无法兑现承诺,生成错误的代码。让模型保持诚实的一个有前景的途径是引入形式化验证:同时生成程序的规范与代码,使代码能够相对于规范被证明正确。不幸的是,现有的大型语言模型在验证式编程方面表现出严重的能力不足。\n\n在本文中,我们展示了如何提升两个预训练模型在 Dafny 这一验证感知语言上的能力。我们使用来自 MBPP 数据集的 178 个问题,提示两个当代模型(GPT-4 和 PaLM-2)合成 Dafny 方法。我们使用了三种不同类型的提示:直接的无上下文提示;包含方法签名和测试用例的签名提示;以及将问题分解为步骤并包含检索增强生成的示例问题与解答的思维链(CoT)提示。我们的结果表明,GPT-4 在这些任务上的表现优于 PaLM-2,并且两个模型在使用检索增强生成的 CoT 提示时表现最佳。GPT-4 能够为 58% 的问题生成经过验证、经人工评估的 Dafny 方法,然而,在使用无上下文提示时,GPT-4 仅完成了 19% 的问题,而在使用签名提示时完成率更低(10%)。因此,我们能够为 MBPP 问题贡献 153 个经过验证的 Dafny 解答,其中 50 个是我们手动编写的,103 个由 GPT-4 合成。\n\n我们的结果表明,形式化程序验证的益处如今已进入代码生成大型语言模型可触及的范围。
英文摘要:
Large language models show great promise in many domains, including programming. A promise is easy to make but hard to keep, and language models often fail to keep their promises, generating erroneous code. A promising avenue to keep models honest is to incorporate formal verification: generating programs' specifications as well as code so that the code can be proved correct with respect to the specifications. Unfortunately, existing large language models show a severe lack of proficiency in verified programming. In this paper, we demonstrate how to improve two pretrained models' proficiency in the Dafny verification-aware language. Using 178 problems from the MBPP dataset, we prompt two contemporary models (GPT-4 and PaLM-2) to synthesize Dafny methods. We use three different types of prompts: a direct Contextless prompt; a Signature prompt that includes a method signature and test cases, and a Chain of Thought (CoT) prompt that decomposes the problem into steps and includes retrieval augmentation generated example problems and solutions. Our results show that GPT-4 performs better than PaLM-2 on these tasks and that both models perform best with the retrieval augmentation generated CoT prompt. GPT-4 was able to generate verified, human-evaluated, Dafny methods for 58% of the problems, however, GPT-4 managed only 19% of the problems with the Contextless prompt, and even fewer (10%) for the Signature prompt. We are thus able to contribute 153 verified Dafny solutions to MBPP problems, 50 that we wrote manually, and 103 synthesized by GPT-4. Our results demonstrate that the benefits of formal program verification are now within reach of code generating large language models...
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。