arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.03997cs.AI
收藏

用大型语言模型引导枚举式程序合成

Guiding Enumerative Program Synthesis with Large Language Models

University of Edinburgh(爱丁堡大学);University of Oxford(牛津大学);University of Innsbruck(因斯布鲁克大学)

Yixuan Li, Julian Parsert, Elizabeth Polgreen

AI总结:

本文提出一种将大型语言模型集成到枚举式程序合成算法中的新方法,在SyGuS基准上显著优于单独使用LLM或枚举合成器及竞赛获胜工具。

摘要:

预训练的大型语言模型(LLMs)正开始主导关于用自然语言规范进行自动代码生成的讨论。相比之下,在具有精确逻辑规范的正式合成领域,性能最好的合成器仍然基于枚举算法。在本文中,我们通过为该领域精心设计一组提示库,评估了LLMs解决正式合成基准测试的能力。当一次性合成失败时,我们提出了一种新颖的枚举式合成算法,该算法将对LLM的调用集成到加权概率搜索中。这使得合成器能够向LLM提供关于枚举器进度的信息,而LLM则能在迭代循环中向枚举器提供语法指导。我们在语法引导合成(SyGuS)竞赛的基准测试上评估了我们的技术。我们发现,GPT-3.5作为正式合成的独立工具,很容易被最先进的正式合成算法超越,但我们将LLM集成到枚举式合成算法中的方法,在性能上显著优于单独的LLM、单独的枚举式合成器以及SyGuS竞赛的获胜工具。

英文摘要:

Pre-trained Large Language Models (LLMs) are beginning to dominate the discourse around automatic code generation with natural language specifications. In contrast, the best-performing synthesizers in the domain of formal synthesis with precise logical specifications are still based on enumerative algorithms. In this paper, we evaluate the abilities of LLMs to solve formal synthesis benchmarks by carefully crafting a library of prompts for the domain. When one-shot synthesis fails, we propose a novel enumerative synthesis algorithm, which integrates calls to an LLM into a weighted probabilistic search. This allows the synthesizer to provide the LLM with information about the progress of the enumerator, and the LLM to provide the enumerator with syntactic guidance in an iterative loop. We evaluate our techniques on benchmarks from the Syntax-Guided Synthesis (SyGuS) competition. We find that GPT-3.5 as a stand-alone tool for formal synthesis is easily outperformed by state-of-the-art formal synthesis algorithms, but our approach integrating the LLM into an enumerative synthesis algorithm shows significant performance gains over both the LLM and the enumerative synthesizer alone and the winning SyGuS competition tool.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑