arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.09342cs.PLcs.AIcs.CLcs.SE
收藏

对用于程序验证的LLM生成的循环不变式进行排序

Ranking LLM-Generated Loop Invariants for Program Verification

Microsoft Research(微软研究院)

Saikat Chakraborty, Shuvendu K. Lahiri, Sarah Fakhoury, Madanlal Musuvathi, Akash Lal, Aseem Rastogi, Aditya Senthilnathan, Rahul Sharma, Nikhil Swamy

AI总结:

针对LLM生成循环不变式需多次采样导致验证器调用过多的问题,提出一种对比排序器对候选不变式重新排序,显著提升了正确不变式的排名并减少了验证器调用次数。

摘要:

归纳循环不变式的合成是实现程序验证自动化的基础。在本研究中,我们观察到大型语言模型(如gpt-3.5或gpt-4)能够在0-shot设置下为某一类程序合成循环不变式,但需要多次采样才能生成正确的不变式。这可能导致大量调用程序验证器来确立一个不变式。为解决此问题,我们提出了一种对LLM生成结果进行重新排序的方法。我们设计了一个排序器,能够根据问题定义区分正确的归纳不变式与错误的尝试。该排序器被优化为对比排序器。实验结果表明,这种重新排序机制显著提升了正确不变式在生成候选结果中的排序,从而大幅减少了对验证器的调用次数。本文的源代码和实验数据可在https://github.com/microsoft/NeuralInvariantRanker获取。

英文摘要:

Synthesizing inductive loop invariants is fundamental to automating program verification. In this work, we observe that Large Language Models (such as gpt-3.5 or gpt-4) are capable of synthesizing loop invariants for a class of programs in a 0-shot setting, yet require several samples to generate the correct invariants. This can lead to a large number of calls to a program verifier to establish an invariant. To address this issue, we propose a {\it re-ranking} approach for the generated results of LLMs. We have designed a ranker that can distinguish between correct inductive invariants and incorrect attempts based on the problem definition. The ranker is optimized as a contrastive ranker. Experimental results demonstrate that this re-ranking mechanism significantly improves the ranking of correct invariants among the generated candidates, leading to a notable reduction in the number of calls to a verifier. The source code and the experimental data for this paper are available in \url{https://github.com/microsoft/NeuralInvariantRanker}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑