arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.08593cs.CLcs.AI
收藏

必要时再调用我:LLM 可在结构化环境中高效且忠实地进行推理

Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments

State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Microsoft(微软)

Sitao Cheng, Ziyuan Zhuang, Yong Xu, Fangkai Yang, Chaoyun Zhang, Xiaoting Qin, Xiang Huang, Ling Chen, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang

AI总结:

本文提出 Readi 框架,让 LLM 在结构化环境中首先生成推理路径并仅在必要时编辑,在多个 KGQA 和 TableQA 数据集上显著超越以往基于 LLM 的方法。

摘要:

大型语言模型(LLM)已在结构化环境(如知识图谱和表格)中的推理任务上展现出潜力。此类任务通常需要多跳推理,即将自然语言表达与环境中的实例进行匹配。以往方法利用 LLM 逐步构建推理路径,LLM 通过与环境逐步交互来调用工具或选择模式。我们提出 Reasoning-Path-Editing(Readi),这是一个新颖的框架,使 LLM 能够在结构化环境中高效且忠实地进行推理。在 Readi 中,LLM 首先根据查询生成一条推理路径,仅在必要时对该路径进行编辑。我们将路径实例化到结构化环境中,并在出现任何错误时提供反馈以编辑路径。在三个 KGQA 数据集和两个 TableQA 数据集上的实验结果表明了 Readi 的有效性,显著超越了以往基于 LLM 的方法(在 WebQSP 上 Hit@1 提升 9.1%,在 MQA-3H 上提升 12.4%,在 WTQ 上提升 9.5%),与最先进的微调方法相当(在 CWQ 上为 67%,在 WebQSP 上为 74.7%),并大幅提升了原始 LLM 的性能(在 CWQ 上提升 14.9%)。我们的代码将在 https://aka.ms/readi 上公开。

英文摘要:

Large Language Models (LLMs) have shown potential in reasoning over structured environments, e.g., knowledge graph and table. Such tasks typically require multi-hop reasoning, i.e., match natural language utterance with instances in the environment. Previous methods leverage LLMs to incrementally build a reasoning path, where the LLMs either invoke tools or pick up schemas by step-by-step interacting with the environment. We propose Reasoning-Path-Editing (Readi), a novel framework where LLMs can efficiently and faithfully reason over structured environments. In Readi, LLMs initially generate a reasoning path given a query, and edit the path only when necessary. We instantiate the path on structured environments and provide feedback to edit the path if anything goes wrong. Experimental results on three KGQA and two TableQA datasets show the effectiveness of Readi, significantly surpassing previous LLM-based methods (by 9.1% Hit@1 on WebQSP, 12.4% on MQA-3H and 9.5% on WTQ), comparable with state-of-the-art fine-tuned methods (67% on CWQ and 74.7% on WebQSP) and substantially boosting the vanilla LLMs (by 14.9% on CWQ). Our code will be available on https://aka.ms/readi.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑