强化微调的幻觉税
The Hallucination Tax of Reinforcement Finetuning
AI总结：
本研究提出强化微调存在“幻觉税”副作用，构建SUM不可解数学数据集验证标准RFT会大幅降低模型拒答率，发现加入10%SUM数据可在极小准确率损失下恢复拒答能力并提升泛化性。
摘要：
强化微调（Reinforcement finetuning, RFT）已成为提升大语言模型（large language models, LLMs）推理能力的标准方法，但其对模型可信度的影响仍未得到充分研究。本研究识别并系统探究了RFT的一个关键副作用，我们将其命名为“幻觉税”：即模型的拒答行为退化，导致其对无法回答的问题自信地输出幻觉性答案。为研究该问题，我们构建了SUM（Synthetic Unanswerable Math，合成不可解数学数据集）——一个高质量的不可解数学问题数据集，用于探测模型通过推理信息不足或模糊来识别不可回答问题的能力。实验结果表明，标准RFT训练可使模型拒答率降低80%以上，显著提升了模型产生幻觉的倾向。我们进一步证实，在RFT训练中仅加入10%的SUM数据，就能大幅恢复模型合理的拒答行为，且在可解任务上的准确率损失极小。关键在于，该方法使大语言模型能够利用推理阶段的计算资源来推理自身的不确定性和知识边界，不仅提升了对域外数学问题的泛化能力，也增强了在事实性问答任务上的表现。
英文摘要：
Reinforcement finetuning (RFT) has become a standard approach for enhancing the reasoning capabilities of large language models (LLMs). However, its impact on model trustworthiness remains underexplored. In this work, we identify and systematically study a critical side effect of RFT, which we term the hallucination tax: a degradation in refusal behavior causing models to produce hallucinated answers to unanswerable questions confidently. To investigate this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of unanswerable math problems designed to probe models' ability to recognize an unanswerable question by reasoning from the insufficient or ambiguous information. Our results show that standard RFT training could reduce model refusal rates by more than 80%, which significantly increases model's tendency to hallucinate. We further demonstrate that incorporating just 10% SUM during RFT substantially restores appropriate refusal behavior, with minimal accuracy trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage inference-time compute to reason about their own uncertainty and knowledge boundaries, improving generalization not only to out-of-domain math problems but also to factual question answering tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
