MR-Align: 为大推理模型引入元推理的事实性对齐
MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Models
AI总结：
MR-ALIGN通过元推理引导的对齐框架提升大型推理模型的事实准确性，通过优化推理过程中的状态转移概率来增强事实性。
摘要：
大型推理模型（LRMs）在复杂推理方面表现出强大的能力，但其在依赖证据的事实性问题上的边际增益有限。我们发现这种限制部分归因于推理-答案命中差距，即模型在推理过程中识别出正确的事实，但未能将其纳入最终回答，从而降低事实准确性。为了解决这个问题，我们提出了MR-ALIGN，一种由元推理引导的对齐框架，该框架在不依赖外部验证器的情况下增强事实性。MR-ALIGN量化模型推理过程中的状态转移概率，并构建一个基于转移的隐式奖励，以在原子推理段落中强化有益的推理模式并抑制缺陷模式。这种重新加权将令牌级信号转换为概率感知的段落分数，鼓励更有利于事实准确性的连贯推理轨迹。在四个事实性问答数据集和一个长文本事实性基准上的实证评估表明，MR-ALIGN在提高准确性和真实性的同时减少了误导性推理。这些结果强调，对推理过程本身的对齐，而不是仅仅对输出进行对齐，对于推进LRMs的事实性至关重要。
英文摘要：
Large reasoning models (LRMs) show strong capabilities in complex reasoning, yet their marginal gains on evidence-dependent factual questions are limited. We find this limitation is partially attributable to a reasoning-answer hit gap, where the model identifies the correct facts during reasoning but fails to incorporate them into the final response, thereby reducing factual fidelity. To address this issue, we propose MR-ALIGN, a Meta-Reasoning informed alignment framework that enhances factuality without relying on external verifiers. MR-ALIGN quantifies state transition probabilities along the model's thinking process and constructs a transition-aware implicit reward that reinforces beneficial reasoning patterns while suppressing defective ones at the atomic thinking segments. This re-weighting reshapes token-level signals into probability-aware segment scores, encouraging coherent reasoning trajectories that are more conducive to factual correctness. Empirical evaluations across four factual QA datasets and one long-form factuality benchmark show that MR-ALIGN consistently improves accuracy and truthfulness while reducing misleading reasoning. These results highlight that aligning the reasoning process itself, rather than merely the outputs, is pivotal for advancing factuality in LRMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
