ARCHE：一种评估大语言模型潜在推理链提取能力的新任务
ARCHE: A Novel Task to Evaluate LLMs on Latent Reasoning Chain Extraction
AI总结：
针对LLMs生成的类推理内容非结构化、难辨其是否理解科学推理范式的问题，本文提出ARCHE任务与RLT表示形式，发布ARCHE Bench数据集及两种评估指标，评估发现现有LLMs在推理链提取的完整性与逻辑性间存在权衡且均未达标。
摘要：
大型语言模型（LLMs）在科学领域的应用日益广泛。虽然它们可以通过思维链提示等方法生成类推理内容，但这些输出通常是非结构化且非正式的，掩盖了模型是否真正理解支撑科学推理的基本推理范式。为解决这一问题，我们引入了一项名为潜在推理链提取（ARCHE）的新任务，要求模型将复杂推理论证分解为标准推理范式的组合，并以推理逻辑树（RLT）的形式呈现。在RLT中，所有推理步骤都被明确归类为皮尔士基本推理模式的三种变体之一：演绎、归纳或溯因。为推进该任务，我们发布了ARCHE Bench基准数据集，该数据集源自70篇《自然通讯》文章，包含超过1900篇参考文献和38000个观点。我们提出了两种逻辑感知评估指标：用于内容完整性的实体覆盖率（EC）和用于逐步逻辑有效性的推理边准确率（REA）。在ARCHE Bench上对10个领先LLMs的评估显示，模型在REA和EC之间存在权衡，且目前没有模型能够提取完整且标准的推理链。这些发现凸显了当前推理模型的能力与科学论证所需严谨性之间的巨大差距。
英文摘要：
Large language models (LLMs) are increasingly used in scientific domains. While they can produce reasoning-like content via methods such as chain-of-thought prompting, these outputs are typically unstructured and informal, obscuring whether models truly understand the fundamental reasoning paradigms that underpin scientific inference. To address this, we introduce a novel task named Latent Reasoning Chain Extraction (ARCHE), in which models must decompose complex reasoning arguments into combinations of standard reasoning paradigms in the form of a Reasoning Logic Tree (RLT). In RLT, all reasoning steps are explicitly categorized as one of three variants of Peirce's fundamental inference modes: deduction, induction, or abduction. To facilitate this task, we release ARCHE Bench, a new benchmark derived from 70 Nature Communications articles, including more than 1,900 references and 38,000 viewpoints. We propose two logic-aware evaluation metrics: Entity Coverage (EC) for content completeness and Reasoning Edge Accuracy (REA) for step-by-step logical validity. Evaluations on 10 leading LLMs on ARCHE Bench reveal that models exhibit a trade-off between REA and EC, and none are yet able to extract a complete and standard reasoning chain. These findings highlight a substantial gap between the abilities of current reasoning models and the rigor required for scientific argumentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
