arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.12342cs.CLcs.AI
收藏

消除推理,通过规划推断:引导大语言模型非线性思维的新框架

Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinking

University of California, San Diego(加州大学圣迭戈分校);University of Pennsylvania(宾夕法尼亚大学);University of Southern California(南加州大学);Yale University(耶鲁大学)

Yongqi Tong, Yifan Wang, Dawei Li, Sizhe Wang, Zi Lin, Simeng Han, Jingbo Shang

AI总结:

提出IEP提示框架,通过规划与自然语言推断的排除过程引导大语言模型非线性思考,并构建含9,115个问题的MARB基准,实验证明其优于CoT且可结合使用。

摘要:

思维链(Chain-of-Thought, CoT)提示及其变体通过模拟人类线性认知和逻辑,探索赋予大语言模型(LLMs)高级推理能力。然而,人类思维是复杂的,同时包含线性和非线性思考。在这项工作中,我们提出了推断排除提示(Inferential Exclusion Prompting, IEP),一种新颖的提示方法,结合了排除和推断的原则,以引导大语言模型进行非线性思考。IEP引导大语言模型进行规划,然后利用自然语言推断(Natural Language Inference, NLI)来推导每个可能解决方案与上下文、常识或事实之间的蕴含关系,从而通过回溯推断获得更广阔的视角。这种前向规划和后向排除的过程使得IEP能够比仅反映线性认知过程的其他基于CoT的方法更好地模拟复杂的人类思维过程。我们进行了一系列实证研究,证实了IEP在各种任务上始终优于CoT。此外,我们观察到将IEP和CoT结合能进一步提升大语言模型在特定任务上的表现,凸显了为大语言模型配备混合逻辑过程的必要性。此外,为了更好地评估人类逻辑中固有的综合特征,我们引入了心智能力推理基准(Mental-Ability Reasoning Benchmark, MARB)。该基准包含六个新颖的子任务,共9,115个问题,其中1,685个问题附有手工制作的推理参考。我们相信IEP和MARB都可以作为揭示大语言模型逻辑和语言推理能力的有前景方向,并推动进一步的进展。MARB将很快在匿名链接上提供。

英文摘要:

Chain-of-Thought(CoT) prompting and its variants explore equipping large language models (LLMs) with high-level reasoning abilities by emulating human-like linear cognition and logic. However, the human mind is complicated and mixed with both linear and nonlinear thinking. In this work, we propose \textbf{I}nferential \textbf{E}xclusion \textbf{P}rompting (IEP), a novel prompting that combines the principles of elimination and inference in order to guide LLMs to think non-linearly. IEP guides LLMs to plan and then utilize Natural Language Inference (NLI) to deduce each possible solution's entailment relation with context, commonsense, or facts, therefore yielding a broader perspective by thinking back for inferring. This forward planning and backward eliminating process allows IEP to better simulate the complex human thinking processes compared to other CoT-based methods, which only reflect linear cognitive processes. We conducted a series of empirical studies and have corroborated that IEP consistently outperforms CoT across various tasks. Additionally, we observe that integrating IEP and CoT further improves the LLMs' performance on certain tasks, highlighting the necessity of equipping LLMs with mixed logic processes. Moreover, to better evaluate comprehensive features inherent in human logic, we introduce \textbf{M}ental-\textbf{A}bility \textbf{R}easoning \textbf{B}enchmark (MARB). The benchmark comprises six novel subtasks with a total of 9,115 questions, among which 1,685 are developed with hand-crafted rationale references. We believe both \textsc{IEP} and \textsc{MARB} can serve as a promising direction for unveiling LLMs' logic and verbal reasoning abilities and drive further advancements. \textsc{MARB} will be available at ~\texttt{anonymity link} soon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑