可信推理：评估与增强LLM中间思维过程中的事实准确性
Trustworthy Reasoning: Evaluating and Enhancing Factual Accuracy in LLM Intermediate Thought Processes
AI总结：
针对LLM中间推理步骤事实错误问题，提出集成事实核查分类器、增强GRPO和激活分析的框架，显著提升事实鲁棒性（最高49.90%），并保持基准性能。
摘要：
我们提出了一种新框架，旨在解决大型语言模型（LLM）中的一个关键漏洞：尽管最终答案正确，但中间推理步骤中普遍存在事实性错误。这一现象在医疗、法律分析和科学研究等高风险领域构成重大风险，因为错误却自信呈现的推理可能误导用户做出危险决策。我们的框架整合了三个核心组件：（1）一个专门的事实核查分类器，基于反事实增强数据进行训练，以检测推理链中细微的事实不一致；（2）一种增强的组相对策略优化（GRPO）强化学习方法，通过多维奖励平衡事实性、连贯性和结构正确性；（3）一种机制可解释性方法，研究推理过程中事实性改进如何在模型激活中体现。跨多个最先进模型的广泛评估揭示了令人担忧的模式：即使是Claude-3.7和GPT-o1等领先模型，其推理事实准确性也分别仅为81.93%和82.57%。我们的方法显著增强了事实鲁棒性（最高提升49.90%），同时在Math-500、AIME-2024和GPQA等具有挑战性的基准上保持或提升了性能。此外，我们的神经激活层面分析提供了可操作的见解，说明事实增强如何在模型架构内重塑推理轨迹，为未来通过激活引导优化明确针对事实鲁棒性的训练方法奠定了基础。
英文摘要：
We present a novel framework addressing a critical vulnerability in Large Language Models (LLMs): the prevalence of factual inaccuracies within intermediate reasoning steps despite correct final answers. This phenomenon poses substantial risks in high-stakes domains including healthcare, legal analysis, and scientific research, where erroneous yet confidently presented reasoning can mislead users into dangerous decisions. Our framework integrates three core components: (1) a specialized fact-checking classifier trained on counterfactually augmented data to detect subtle factual inconsistencies within reasoning chains; (2) an enhanced Group Relative Policy Optimization (GRPO) reinforcement learning approach that balances factuality, coherence, and structural correctness through multi-dimensional rewards; and (3) a mechanistic interpretability method examining how factuality improvements manifest in model activations during reasoning processes. Extensive evaluation across multi state-of-the-art models reveals concerning patterns: even leading models like Claude-3.7 and GPT-o1 demonstrate reasoning factual accuracy of only 81.93% and 82.57% respectively. Our approach significantly enhances factual robustness (up to 49.90% improvement) while maintaining or improving performance on challenging benchmarks including Math-500, AIME-2024, and GPQA. Furthermore, our neural activation-level analysis provides actionable insights into how factual enhancements reshape reasoning trajectories within model architectures, establishing foundations for future training methodologies that explicitly target factual robustness through activation-guided optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
