剖析大语言模型中的逻辑推理：一项细粒度评估与监督研究
Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study
AI总结：
提出FineLogic细粒度评估框架，从准确率、逐步合理性和表示探测三维度分析微调监督格式对LLM逻辑推理的影响，发现自然语言监督利于泛化，符号监督利于结构严谨性。
摘要：
逻辑推理是大语言模型（LLMs）的一项核心能力，然而现有仅依赖最终答案准确率的基准无法捕捉推理过程的质量。为解决这一问题，我们提出了FineLogic，一个细粒度评估框架，从三个维度评估逻辑推理：整体准确率、逐步合理性和表示层探测。利用该框架，我们对微调中不同监督格式如何塑造推理能力进行了全面研究。我们在四种监督风格上微调LLMs：一种自然语言风格和三种符号变体。我们发现了一个关键权衡：自然语言监督在分布外和长链问题的泛化上表现出色，而符号监督在灌输结构健全的原子推理步骤方面更优。此外，我们的探测分析表明，微调主要优化模型的逐步生成过程，而非提升其早期收敛到答案的能力。综合来看，我们的框架和分析为评估和改进LLMs中的逻辑推理提供了更严谨的视角。代码可在https://github.com/YujunZhou/FineLogic获取。
英文摘要：
Logical reasoning is a core capability for large language models (LLMs), yet existing benchmarks that rely solely on final-answer accuracy fail to capture the quality of the reasoning process. To address this, we introduce FineLogic, a fine-grained evaluation framework that assesses logical reasoning across three dimensions: overall accuracy, stepwise soundness, and representation-level probing. Leveraging this framework, we conduct a comprehensive study on how different supervision formats in fine-tuning shape reasoning abilities. We fine-tune LLMs on four supervision styles: one in natural language and three symbolic variants. We find a key trade-off: natural language supervision excels at generalization to out-of-distribution and long-chain problems, whereas symbolic supervision is superior at instilling structurally sound, atomic reasoning steps. Furthermore, our probing analysis indicates that fine-tuning primarily refines the model's step-by-step generation process, rather than improving its ability to converge on an answer early. Together, our framework and analysis provide a more rigorous lens for evaluating and improving logical reasoning in LLMs. The code is available at https://github.com/YujunZhou/FineLogic.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
