迈向 LogiGLUE：语言模型逻辑推理能力分析的简要综述与基准
Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models
AI总结：
本文提出 LogiGLUE 基准，涵盖 24 个数据集和演绎、溯因、归纳三类推理，并训练 LogiT5 模型，发现大语言模型在溯因推理上表现最好、归纳推理最差，为提升逻辑推理能力提供分析基础。
摘要：
逻辑推理对人类而言是基础能力，但在人工智能领域仍构成重大挑战。最初，研究者使用知识表示与推理（KR）系统，这些系统难以扩展且需要大量非平凡的人工工作。近年来，大语言模型（LLMs）的出现展现出克服形式化知识表示（KR）系统诸多局限的能力。因此，人们越来越关注通过自然语言利用大语言模型进行逻辑推理。本工作旨在通过简要回顾该领域最新进展来理解大语言模型在逻辑推理方面的熟练程度，重点关注逻辑推理数据集、任务以及利用大语言模型进行推理所采用的方法。为了提供全面分析，我们编制了一个名为 LogiGLUE 的基准，包含 24 个多样化的数据集，涵盖演绎推理、溯因推理和归纳推理。以 LogiGLUE 为基础，我们训练了一个指令微调语言模型，得到 LogiT5。我们研究了单任务训练、多任务训练以及“思维链”知识蒸馏微调技术，以评估模型在不同逻辑推理类别上的表现。我们还使用 LogiGLUE 评估了多种大语言模型，结果表明大语言模型在溯因推理上表现最佳，其次是演绎推理，而在归纳推理上效果最差。我们旨在揭示大语言模型逻辑推理能力提升的潜力与可能路径，为这一关键领域更先进、更细致的发展铺平道路。
英文摘要：
Logical reasoning is fundamental for humans yet presents a substantial challenge in the domain of Artificial Intelligence. Initially, researchers used Knowledge Representation and Reasoning (KR) systems that did not scale and required non-trivial manual effort. Recently, the emergence of large language models (LLMs) has demonstrated the ability to overcome various limitations of formal Knowledge Representation (KR) systems. Consequently, there's a growing interest in using LLMs for logical reasoning via natural language. This work strives to understand the proficiency of LLMs in logical reasoning by offering a brief review of the latest progress in this area; with a focus on the logical reasoning datasets, tasks, and the methods adopted to utilize LLMs for reasoning. To offer a thorough analysis, we have compiled a benchmark titled LogiGLUE. This includes 24 varied datasets encompassing deductive, abductive, and inductive reasoning. Utilizing LogiGLUE as a foundation, we have trained an instruction fine-tuned language model, resulting in LogiT5. We study single-task training, multi-task training, and "chain-of-thought" knowledge distillation fine-tuning technique to assess the performance of model across the different logical reasoning categories. We also assess various LLMs using LogiGLUE, and the findings indicate that LLMs excel most in abductive reasoning, followed by deductive reasoning, while they are least effective at inductive reasoning. We aim to shed light on the capabilities and potential pathways for enhancing logical reasoning proficiency in LLMs, paving the way for more advanced and nuanced developments in this critical field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
