学习停止而不学习停止：自监督置信度训练提高推理效率

中文摘要 AI 辅助

推理模型通常生成非常长的推理轨迹，使得推理计算成本高昂。现有方法通常通过推理时的早停机制或在训练中明确鼓励更短的推理（例如通过带长度惩罚的强化学习）来提高效率。我们表明，实质性的效率提升可以来自另一种监督：\textit{置信度}。通过自监督程序，我们仅使用600个训练问题，对推理模型进行微调，使其在自身推理轨迹的中间点预测对答案的置信度。置信度仅用作训练目标：损失中不包含任何关于推理长度、效率或停止的目标。在推理时，微调后的模型使用标准生成程序，没有置信度引出或早停机制。尽管如此，自监督置信度微调使推理更高效，在Gemma、Qwen、Nemotron和GPT-OSS模型上，针对数学、科学和编码推理基准，在匹配准确率下生成的token最多减少25%，效率提升与明确优化更短推理的方法相当。对推理片段的分析进一步表明，置信度监督在很大程度上保留了基础模型的高层推理组成，而不是选择性地抑制特定行为。我们的结果表明，高效推理可能作为学习元认知信号的下游结果而出现，而无需直接优化。

英文摘要

Reasoning models often generate very long reasoning traces, making inference computationally expensive. Existing approaches typically improve efficiency either through inference-time early-stopping mechanisms or by explicitly encouraging shorter reasoning during training, for example through reinforcement learning with length penalties. We show that substantial efficiency gains can instead emerge from a different kind of supervision: \textit{confidence}. Using a self-supervised procedure, we fine-tune reasoning models to predict their confidence in the answer at intermediate points along their own reasoning trajectories using only 600 training problems. Confidence is used only as a training target: the loss contains no objective for reasoning length, efficiency, or stopping. At inference, the fine-tuned models use the standard generation procedure, with no confidence elicitation or early-stopping mechanism. Despite this, self-supervised confidence fine-tuning makes reasoning more efficient, reducing generated tokens by up to 25\% at matched accuracy across Gemma, Qwen, Nemotron, and GPT-OSS models on mathematical, scientific, and coding reasoning benchmarks, with efficiency gains comparable to methods that explicitly optimize for shorter reasoning. Analysis of reasoning episodes further shows that confidence supervision largely preserves the base models' high-level reasoning composition rather than selectively suppressing particular behaviors. Our results suggest that efficient reasoning may emerge as a downstream consequence of learning metacognitive signals, without being directly optimized.

发表机构

• University of Maryland（马里兰大学）
• AI Foundations, Capital One（第一资本人工智能基础部）

机构由 AI 辅助整理，请以论文原文为准。