首次尝试至关重要:重新审视反思在推理模型中的作用
First Try Matters: Revisiting the Role of Reflection in Reasoning Models
AI总结:
本文通过分析八个推理模型在五个数学数据集上的生成轨迹,发现反思多为确认性且很少改变初始答案,并提出问题感知早停与动态截断反思方法,以减少24.5%推理令牌且准确率仅降2.9%。
摘要:
大型语言模型近期在推理能力上展现出显著进步,这通常归因于它们生成更长的思维链并进行反思性推理的能力。然而,反思对性能提升的贡献仍不明确。本文系统分析了八个推理模型在五个数学数据集上的生成轨迹。我们聚焦于反思行为,即模型已产生答案但在最终确定输出前继续反思的情况。分析显示,反思主要是确认性的,很少改变模型的初始答案,这一模式在模型和数据集间保持一致。为理解反思在训练中的作用,我们构建了具有不同反思步骤数量的监督微调(SFT)数据集。我们观察到,在具有更多反思步骤的生成轨迹上训练模型,主要提升的是首次答案的正确性,而非通过反思纠正初始错误答案的能力。这促使我们提出一种问题感知的早停方法,通过在生成几个合理候选答案后停止推理过程来提升推理时的令牌效率,从而减少不必要的反思步骤。基于此,我们进一步提出在生成过程中出现候选答案后动态截断反思,这使五个数学数据集上的推理令牌减少24.5%,准确率仅下降2.9%。
英文摘要:
Large language models have recently demonstrated significant gains in reasoning ability, often attributed to their capacity to generate longer chains of thought and engage in reflective reasoning. However, the contribution of reflections to performance improvement remains unclear. In this paper, we systematically analyze the rollouts of eight reasoning models on five mathematical datasets. We focus on reflective behaviours where the model has already produced an answer but continues reflecting before finalizing its output. Our analysis reveals that reflections are predominantly confirmatory and rarely alter the model's initial answer, a pattern consistent across models and datasets. To understand the role of reflections in training, we construct supervised fine-tuning (SFT) datasets with varying amounts of reflection steps. We observe that training models on rollouts with more reflection steps primarily enhances first-answer correctness rather than the ability to correct initially wrong answers through reflections. This motivates us to propose a question-aware early-stopping method that enhances inference-time token efficiency by stopping the reasoning process once a few plausible candidate answers are generated, thereby reducing unnecessary reflection steps. Motivated by this, we further propose to dynamically truncate the reflections after a candidate answer has appeared during generation, which reduces reasoning tokens by 24.5% across five mathematical datasets, within a 2.9% drop in accuracy.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。