解耦逻辑：上下文在大型语言模型推理能力中的作用
Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities
AI总结：
本研究通过对比抽象与上下文化逻辑问题，系统解耦纯逻辑推理与文本理解，探究LLMs在保持逻辑结构不变时的真正推理能力及微调泛化潜力，并构建了包含4个难度级别、12个领域的命题推理数据集。
摘要：
本研究旨在通过考察涵盖多个领域的抽象与上下文化逻辑问题之间的对比，系统性地解耦纯逻辑推理与文本理解。我们探讨了当下层逻辑结构保持不变时，LLMs（大型语言模型）在不同领域是否展现出真正的推理能力。我们聚焦于两个主要问题：(1) 脱离实际场景中的上下文支持，仅凭抽象逻辑问题能否准确基准测试LLM在真实世界场景中的推理能力？(2) 在抽象逻辑问题上对LLMs进行微调，能否泛化到上下文化逻辑问题，反之亦然？为研究这些问题，我们聚焦于标准命题逻辑，具体为命题演绎与溯因逻辑推理。特别地，我们基于Wikipedia的分类构建了演绎与溯因推理的实例化数据集，包含4个难度级别，涵盖12个不同类别或领域。我们的实验旨在为解耦逻辑推理中的上下文、LLMs的真正推理能力及其泛化潜力提供深刻见解。代码和数据集可在以下地址获取：https://github.com/agiresearch/ContextHub。
英文摘要：
This study intends to systematically disentangle pure logic reasoning and text understanding by investigating the contrast across abstract and contextualized logical problems from a comprehensive set of domains. We explore whether LLMs demonstrate genuine reasoning capabilities across various domains when the underlying logical structure remains constant. We focus on two main questions (1) Can abstract logical problems alone accurately benchmark an LLM's reasoning ability in real-world scenarios, disentangled from contextual support in practical settings? (2) Does fine-tuning LLMs on abstract logic problem generalize to contextualized logic problems and vice versa? To investigate these questions, we focus on standard propositional logic, specifically propositional deductive and abductive logic reasoning. In particular, we construct instantiated datasets for deductive and abductive reasoning with 4 levels of difficulty, encompassing 12 distinct categories or domains based on the categorization of Wikipedia. Our experiments aim to provide insights into disentangling context in logical reasoning and the true reasoning capabilities of LLMs and their generalization potential. The code and dataset are available at: https://github.com/agiresearch/ContextHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
