ReFIne：兼具可靠性、忠实性与可解释性的可信赖大型推理模型框架
ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability
AI总结：
该研究提出 ReFIne 框架，结合监督微调与 GRPO 训练 Qwen3 模型，使其在数学基准上的推理轨迹可解释性、忠实性、可靠性均显著提升，推动推理模型向多维度可信赖性优化发展。
摘要：
近期长思维链（CoT）推理的进展主要聚焦于答案准确性和 token 效率，却忽视了对可信赖性至关重要的方面。我们认为，可用的推理系统必须具备可信赖性，其核心特征包括可解释性、忠实性与可靠性。为此，我们提出 ReFIne，这是一个将监督微调与 GRPO 相结合的新型训练框架，旨在引导模型实现三个目标：（i）生成带有高层规划的结构化、基于标签的推理轨迹，提升可解释性，使其更易于人类理解；（ii）通过明确披露每个解决方案的决定性信息并提供一致的截面引用，增强忠实性；（iii）对推导的合理性及最终答案的置信度进行自我评估，提升可靠性。我们将 ReFIne 应用于多个规模的 Qwen3 模型（1.7B/4B/8B），并在不同难度的数学基准上进行评估。实验结果显示，ReFIne 模型生成的推理轨迹更清晰、结构更合理（可解释性提升 44.0%），能更忠实地呈现其底层决策过程（忠实性提升 18.8%），并提供有价值的置信度估计（可靠性提升 42.4%）。这些发现凸显了一个被忽视但重要的方向：推理模型不仅应针对准确性进行优化，还应针对更广泛的可信赖性维度进行优化。我们的代码可在以下网址获取：this https URL
英文摘要：
Recent advances in long chain-of-thought (CoT) reasoning have largely prioritized answer accuracy and token efficiency, while overlooking aspects critical to trustworthiness. We argue that usable reasoning systems must be trustworthy, characterized by three properties: interpretability, faithfulness, and reliability. To this end, we propose ReFIne, a new training framework that integrates supervised fine-tuning with GRPO to encourage models to: (i) improve interpretability by producing structured, tag-based traces with high-level planning that are easier for humans to follow; (ii) enhance faithfulness by explicitly disclosing the decisive information guiding each solution, with consistent cross-section references; and (iii) promote reliability by providing self-assessments of both the derivation's soundness and the confidence of the final answer. We apply ReFIne to the Qwen3 models at multiple scales (1.7B/4B/8B) and evaluate across mathematical benchmarks of varying difficulty. Our experimental results show that ReFIne models generate clearer and better-structured reasoning traces (interpretability +44.0%), more faithfully expose their underlying decision process (faithfulness +18.8%), and offer informative confidence estimates (reliability +42.4%). These findings highlight an overlooked but important direction: reasoning models should be optimized not only for accuracy, but also for broader dimensions of trustworthiness. Our code is available at: https://github.com/Trustworthy-ML-Lab/Training_Trustworthy_LRM_with_Refine
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
