VerMCTS:使用验证器、大型语言模型和树搜索合成多步程序
VerMCTS: Synthesizing Multi-Step Programs using a Verifier, a Large Language Model, and Tree Search
AI总结:
本文提出VerMCTS,结合验证器与LLM改进蒙特卡洛树搜索,在Dafny和Coq中合成已验证的多步程序,平均pass@5000提升超30%。
摘要:
大型语言模型(LLM)能够生成有用的代码,但通常生成的代码在可靠性方面无法得到保证。在本文中,我们提出了VerMCTS,一种通过生成Dafny和Coq中的已验证程序来解决此问题的方法。VerMCTS将逻辑验证器与LLM结合,以指导改进的蒙特卡洛树搜索(MCTS)。该方法利用验证器在搜索算法内部获得中间反馈,通过逐步检查部分程序来估计价值函数的上界。为了衡量VerMCTS的性能,我们在Dafny和Coq中开发了一套新的多步验证编程问题。在pass@T(一种新指标,根据从LLM采样的T个token预算计算通过率)方面,与从基础语言模型重复采样相比,VerMCTS在整个套件上的平均pass@5000绝对提高了超过30%。我们的代码和基准可在https://github.com/namin/llm-verified-with-monte-carlo-tree-search获取。
英文摘要:
Large Language Models (LLMs) can generate useful code, but often the code they generate cannot be trusted to be sound. In this paper, we present VerMCTS, an approach to begin to resolve this issue by generating verified programs in Dafny and Coq. VerMCTS uses a logical verifier in concert with an LLM to guide a modified Monte Carlo Tree Search (MCTS). This approach leverages the verifier to gain intermediate feedback inside the search algorithm by checking partial programs at each step to estimate an upper bound on the value function. To measure the performance of VerMCTS, we develop a new suite of multi-step verified programming problems in Dafny and Coq. In terms of pass@T, a new metric which computes the pass rate given a budget of T tokens sampled from the LLM, VerMCTS leads to more than a 30% absolute increase in average pass@5000 across the suite over repeated sampling from the base language model. Our code and benchmarks are available at https://github.com/namin/llm-verified-with-monte-carlo-tree-search .
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。