arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.16097cs.AIcs.LOcs.SE
收藏

语言模型能否模拟求解器?基于LLM的逻辑代码仿真

Can Language Models Pretend Solvers? Logic Code Simulation with LLMs

Shanghai Jiao Tong University(上海交通大学);Shanghai Polytechnic University(上海第二工业大学)

Minyu Chen, Guoqiang Li, Ling-I Wu, Ruibang Liu, Yuxin Su, Xi Chang, Jianxin Xue

AI总结:

本文研究LLM的逻辑代码仿真能力,提出双路径思考技术DCoL,在GPT-4-Turbo上实现准确率提升7.06%的最优性能。

摘要:

基于Transformer的大语言模型(LLMs)在解决逻辑问题方面展现出显著潜力。借助LLMs在代码相关活动中的强大能力,近期已提出若干利用逻辑求解器进行逻辑推理的框架。现有研究主要将LLMs视为自然语言逻辑求解器或翻译器,而它们作为逻辑代码解释器和执行器的角色受到的关注有限。本研究探讨一个新的方面,即逻辑代码仿真,该任务要求LLMs模拟逻辑求解器以预测逻辑程序的运行结果。为深入研究这一新任务,我们提出三个研究问题:LLMs能否高效模拟逻辑代码的输出?逻辑代码仿真会带来哪些优势?又存在哪些陷阱?为回答这些问题,我们构建了三个专门针对逻辑代码仿真任务的新数据集,并进行充分实验以建立LLMs在代码仿真中的基线性能。随后,我们提出一种开创性的基于LLM的代码仿真技术——逻辑双链(DCoL)。该技术倡导LLMs采用双路径思考方式,与其他LLM提示策略相比已展现出最先进的性能,在使用GPT-4-Turbo时准确率显著提升7.06%。

英文摘要:

Transformer-based large language models (LLMs) have demonstrated significant potential in addressing logic problems. capitalizing on the great capabilities of LLMs for code-related activities, several frameworks leveraging logical solvers for logic reasoning have been proposed recently. While existing research predominantly focuses on viewing LLMs as natural language logic solvers or translators, their roles as logic code interpreters and executors have received limited attention. This study delves into a novel aspect, namely logic code simulation, which forces LLMs to emulate logical solvers in predicting the results of logical programs. To further investigate this novel task, we formulate our three research questions: Can LLMs efficiently simulate the outputs of logic codes? What strength arises along with logic code simulation? And what pitfalls? To address these inquiries, we curate three novel datasets tailored for the logic code simulation task and undertake thorough experiments to establish the baseline performance of LLMs in code simulation. Subsequently, we introduce a pioneering LLM-based code simulation technique, Dual Chains of Logic (DCoL). This technique advocates a dual-path thinking approach for LLMs, which has demonstrated state-of-the-art performance compared to other LLM prompt strategies, achieving a notable improvement in accuracy by 7.06% with GPT-4-Turbo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑