LogicBench：迈向大语言模型逻辑推理能力的系统化评估
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
AI总结：
本文系统评估大语言模型的逻辑推理能力，提出覆盖命题、一阶和非单调逻辑共25种推理模式的LogicBench数据集，并发现现有LLMs在复杂推理与否定场景中表现不佳。
摘要：
近期开发的大语言模型（LLMs）已被证明在广泛的语言理解任务上表现极为出色。但是，它们真的能对自然语言进行“推理”吗？这一问题已受到大量研究关注，许多推理技能（如常识推理、数值推理和定性推理）都已被研究。然而，与“逻辑推理”相关的关键技能仍未得到充分探索。现有研究在考察LLMs的这一推理能力时，仅关注命题逻辑和一阶逻辑中的少数推理规则（如肯定前件和否定后件）。针对上述局限，我们对LLMs在涵盖命题逻辑、一阶逻辑和非单调逻辑的25种不同推理模式上的逻辑推理能力进行了全面评估。为了实现系统化评估，我们引入了LogicBench，这是一个专注于使用单一推理规则的自然语言问答数据集。我们使用思维链提示，对GPT-4、ChatGPT、Gemini、Llama-2和Mistral等一系列LLMs进行了详细分析。实验结果表明，现有LLMs在LogicBench上表现不佳；尤其是它们在涉及复杂推理和否定的实例上存在困难。此外，它们有时会忽略得出正确结论所必需的上下文信息。我们相信，我们的工作和发现有助于未来评估和增强LLMs逻辑推理能力的研究。数据和代码可在https://github.com/Mihir3009/LogicBench获取。
英文摘要：
Recently developed large language models (LLMs) have been shown to perform remarkably well on a wide range of language understanding tasks. But, can they really "reason" over the natural language? This question has been receiving significant research attention and many reasoning skills such as commonsense, numerical, and qualitative have been studied. However, the crucial skill pertaining to 'logical reasoning' has remained underexplored. Existing work investigating this reasoning ability of LLMs has focused only on a couple of inference rules (such as modus ponens and modus tollens) of propositional and first-order logic. Addressing the above limitation, we comprehensively evaluate the logical reasoning ability of LLMs on 25 different reasoning patterns spanning over propositional, first-order, and non-monotonic logics. To enable systematic evaluation, we introduce LogicBench, a natural language question-answering dataset focusing on the use of a single inference rule. We conduct detailed analysis with a range of LLMs such as GPT-4, ChatGPT, Gemini, Llama-2, and Mistral using chain-of-thought prompting. Experimental results show that existing LLMs do not fare well on LogicBench; especially, they struggle with instances involving complex reasoning and negations. Furthermore, they sometimes overlook contextual information necessary for reasoning to arrive at the correct conclusion. We believe that our work and findings facilitate future research for evaluating and enhancing the logical reasoning ability of LLMs. Data and code are available at https://github.com/Mihir3009/LogicBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
