RFEval: 在大推理模型中评估推理忠实性下的反事实推理干预基准测试
RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models
AI总结：
RFEval通过反事实推理干预评估大推理模型的推理忠实性，发现49.7%的输出不忠实，主要源于立场不一致，强调准确性不能作为忠实性的可靠代理。
摘要：
大推理模型（LRMs）表现出强大的性能，但往往生成的推理过程听起来合理却未能反映其真实的决策过程，这削弱了可靠性和信任度。我们引入了一个正式的推理忠实性框架，其定义由两个可测试的条件组成：立场一致性（一种将推理与答案联系起来的连贯立场）和因果影响（在输出层面的干预下，陈述的推理因果驱动答案），并且明确与准确性分离。为了将这一框架具体化，我们提出了RFEval，一个包含7186个实例的七个任务基准测试，通过受控的输出层面反事实干预来探测忠实性。评估十二个开源LRMs，我们发现49.7%的输出不忠实，主要来自立场不一致。失败集中在脆弱、收敛的领域，如数学和代码，并且更与训练后制度相关，而不是规模：在同一家族中的消融实验表明，在监督微调之上添加当前RL风格的目标可以减少推理忠实性，即使准确性得以保持。关键的是，准确性既不是充分的也不是可靠的忠实性代理：在控制模型和任务后，准确性与忠实性的联系是弱且统计上不显著的。我们的工作建立了一种严格的方法来审计LRM的可靠性，并表明可信的AI需要不仅优化正确的结果，还要优化推理过程的结构完整性。我们的代码和数据集可在项目页面上找到：https://aidaslab.github.io/RFEval/
英文摘要：
Large Reasoning Models (LRMs) exhibit strong performance, yet often produce rationales that sound plausible but fail to reflect their true decision process, undermining reliability and trust. We introduce a formal framework for reasoning faithfulness, defined by two testable conditions: stance consistency (a coherent stance linking reasoning to answer) and causal influence (the stated reasoning causally drives the answer under output-level interventions), explicitly decoupled from accuracy. To operationalize this, we present RFEval, a benchmark of 7,186 instances across seven tasks that probes faithfulness via controlled, output-level counterfactual interventions. Evaluating twelve open-source LRMs, we find unfaithfulness in 49.7% of outputs, predominantly from stance inconsistency. Failures are concentrated in brittle, convergent domains such as math and code, and correlate more with post-training regimes than with scale: within-family ablations indicate that adding current RL-style objectives on top of supervised fine-tuning can reduce reasoning faithfulness, even when accuracy is maintained. Crucially, accuracy is neither a sufficient nor a reliable proxy for faithfulness: once controlling for model and task, the accuracy-faithfulness link is weak and statistically insignificant. Our work establishes a rigorous methodology for auditing LRM reliability and shows that trustworthy AI requires optimizing not only for correct outcomes but also for the structural integrity of the reasoning process. Our code and dataset can be found at project page: https://aidaslab.github.io/RFEval/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
