arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.18327cs.CLcs.AI
收藏

$\forall$uto$\exists$val:形式综合与解释任务中LLM的自主评估

$\forall$uto$\exists$val: Autonomous Assessment of LLMs in Formal Synthesis and Interpretation Tasks

School of Computing and Augmented Intelligence(计算与增强智能学院);Arizona State University(亚利桑那州立大学)

Rushang Karia, Daniel Bramblett, Daksh Dobhal, Pulkit Verma, Siddharth Srivastava

AI总结:

本文提出$\forall$uto$\exists$val,利用上下文无关文法动态生成分布外数据,结合形式验证器进行闭环测试,自主评估LLM在形式语法与自然语言互译任务中的能力,实验表明现有SOTA模型难以胜任。

摘要:

本文提出了$\forall$uto$\exists$val,一种用于扩展LLM评估的新方法,该方法涉及将形式语法(如一阶逻辑、正则表达式等)翻译为自然语言(解释)或反之(编译),从而促进其在生成/解释程序逻辑和控制流等应用中的使用。现有在这些领域进行LLM评估的方法需要耗费大量人力创建地面真值,而地面真值的可用性破坏了训练集和测试集的分离。此外,此类数据集通常包含相对较少的、手工编码的测试用例,LLM的准确性在这些用例上被确定,因此不足以确定其生成输出的安全性或正确性。我们引入了一种新方法,利用上下文无关文法(CFG)动态生成分布外数据集,并使用形式验证器对LLM能力进行闭环测试,以确保LLM输出的正确性,无需任何人工干预。我们将数据集和基准以开源代码形式发布在\url{https://github.com/AAIR-lab/auto-llm-assessment}。我们还对几种SOTA闭源和开源LLM进行了评估,以展示该范式的可行性和可扩展性。我们的实验表明,SOTA LLM无法充分解决形式翻译任务。

英文摘要:

This paper presents $\forall$uto$\exists$val, a new approach for scaling LLM assessment in translating formal syntax -- such as first-order logic, regular expressions, etc -- to natural language (interpretation) or vice versa (compilation), thereby facilitating their use in applications such as generating/explaining logic and control flow for programs etc. Existing approaches for LLM assessment in these areas require labor-intensive ground-truth creation, the availability of which undermines the separation of training and test sets. Furthermore, such datasets typically include relatively few hand-coded test cases over which LLM accuracy is determined, thus making them inadequate for determining the safety or correctness of their generated outputs. We introduce a new approach that utilizes context-free grammars (CFGs) to generate out-of-distribution datasets on the fly and perform closed-loop testing of LLM capabilities using formal verifiers to guarantee the correctness of LLM outputs without any human intervention. We release our dataset and benchmark as open-source code at \url{https://github.com/AAIR-lab/auto-llm-assessment}. We also conduct an assessment of several SOTA closed and open-source LLMs to showcase the feasibility and scalability of this paradigm. Our experiments reveal that SOTA LLMs are unable to solve the formal translation task adequately.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑