Multi-LogiEval：面向评估大语言模型的多步逻辑推理能力
Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models
AI总结：
针对现有逻辑推理基准规则有限、缺非单调推理评估的问题，提出覆盖三类逻辑、多规则多深度的Multi-LogiEval数据集，评估发现LLM性能随推理深度显著下降，为相关研究提供支撑。
摘要：
随着大语言模型（LLMs）在自然语言理解任务中持续展现出优异性能，衡量其类人多步逻辑推理能力成为迫切需求。现有逻辑推理评估基准通常主要关注简单的单步推理，或是仅包含有限推理规则集的多步推理。此外，用于评估非单调推理的数据集存在缺失，这是一项关键空白，因为非单调推理与类人推理的契合度更高。为解决这些局限，我们提出Multi-LogiEval，这是一个综合性评估数据集，涵盖具备多种推理规则与推理深度的多步逻辑推理任务。Multi-LogiEval覆盖三类逻辑——命题逻辑、一阶逻辑与非单调逻辑，包含30余种推理规则及其60余种不同深度的组合。借助该数据集，我们采用零样本思维链（zero-shot chain-of-thought）方法，对GPT-4、ChatGPT、Gemini-Pro、Yi、Orca和Mistral等一系列LLMs开展评估。实验结果表明，随着推理步骤/深度增加，LLMs的性能出现显著下降（深度1时平均准确率约为68%，深度5时降至约43%）。我们进一步对LLMs生成的推理链展开深入分析，揭示了若干重要发现。我们认为Multi-LogiEval将推动未来评估与增强LLMs逻辑推理能力的研究。数据集可在https://github.com/Mihir3009/Multi-LogiEval获取。
英文摘要：
As Large Language Models (LLMs) continue to exhibit remarkable performance in natural language understanding tasks, there is a crucial need to measure their ability for human-like multi-step logical reasoning. Existing logical reasoning evaluation benchmarks often focus primarily on simplistic single-step or multi-step reasoning with a limited set of inference rules. Furthermore, the lack of datasets for evaluating non-monotonic reasoning represents a crucial gap since it aligns more closely with human-like reasoning. To address these limitations, we propose Multi-LogiEval, a comprehensive evaluation dataset encompassing multi-step logical reasoning with various inference rules and depths. Multi-LogiEval covers three logic types--propositional, first-order, and non-monotonic--consisting of more than 30 inference rules and more than 60 of their combinations with various depths. Leveraging this dataset, we conduct evaluations on a range of LLMs including GPT-4, ChatGPT, Gemini-Pro, Yi, Orca, and Mistral, employing a zero-shot chain-of-thought. Experimental results show that there is a significant drop in the performance of LLMs as the reasoning steps/depth increases (average accuracy of ~68% at depth-1 to ~43% at depth-5). We further conduct a thorough investigation of reasoning chains generated by LLMs which reveals several important findings. We believe that Multi-LogiEval facilitates future research for evaluating and enhancing the logical reasoning ability of LLMs. Data is available at https://github.com/Mihir3009/Multi-LogiEval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
