arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.11442cs.CL
收藏

大语言模型能否基于规则推理?用于压力测试与改进大语言模型的逻辑脚手架方法

Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs

Fudan University(复旦大学);University of Washington(华盛顿大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Southern California(南加州大学)

Siyuan Wang, Zhongyu Wei, Yejin Choi, Xiang Ren

AI总结:

本研究针对大语言模型推理规则掌握不足的问题,提出逻辑脚手架框架并构建规则库ULogic,发现LLMs在复杂规则上与人类有差距,经规则蒸馏的推理引擎可有效提升其逻辑推理能力。

摘要:

大语言模型(LLMs)已在各类推理任务上取得了媲美人类的出色表现,但它们对底层推理规则的掌握仍不及人类水平。为探究这一问题,我们提出逻辑脚手架推理规则生成框架,构建了包含五个领域的基础规则与组合规则的推理规则库ULogic。我们在一个规则子集上对GPT系列模型进行分析,发现LLMs的逻辑理解能力与人类表现存在显著差距,尤其是在带有特定偏差模式的组合型、结构复杂规则上表现更差。我们进一步将这些规则蒸馏到一个更小的推理引擎中,用于灵活生成规则并增强下游推理能力。通过多裁判评估,该推理引擎被证明可有效生成准确、复杂且抽象的结论与前提,并提升各类常识推理任务的表现。总体而言,本研究揭示了LLMs在掌握推理规则方面的局限性,并提出了增强其逻辑推理能力的方向。代码与数据可在https://github.com/SiyuanWangw/ULogic获取。

英文摘要:

Large language models (LLMs) have achieved impressive human-like performance across various reasoning tasks. However, their mastery of underlying inferential rules still falls short of human capabilities. To investigate this, we propose a logic scaffolding inferential rule generation framework, to construct an inferential rule base, ULogic, comprising both primitive and compositional rules across five domains. Our analysis of GPT-series models over a rule subset reveals significant gaps in LLMs' logic understanding compared to human performance, especially in compositional and structural complex rules with certain bias patterns. We further distill these rules into a smaller-scale inference engine for flexible rule generation and enhancing downstream reasoning. Through a multi-judger evaluation, our inference engine proves effective in generating accurate, complex and abstract conclusions and premises, and improve various commonsense reasoning tasks. Overall, our work sheds light on LLMs' limitations in grasping inferential rule and suggests ways to enhance their logical reasoning abilities~\footnote{Code and data are available at \url{https://github.com/SiyuanWangw/ULogic}.}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑