逐步惩罚以实现高效推理链
Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning
AI总结：
SWAP通过逐步自适应惩罚减少推理链长度并提升准确性。
摘要：
大型推理模型通过增加测试时的计算量而得到改进，但往往过度思考，产生不必要的长推理链，增加成本而不提升准确性。先前的强化学习方法通常依赖于单一结果奖励和轨迹级长度惩罚，无法区分必要的和冗余的推理步骤，因此导致压缩效果粗劣。尽管最近的工作纳入了步骤级信号，如离线剪枝、监督数据构建或基于验证者的中间奖励，但在RL中很少将推理长度作为显式的步骤级优化目标。我们提出了逐步自适应惩罚（SWAP），一个细粒度框架，根据内在贡献在步骤间分配长度减少。我们从模型的on-policy日志概率改进估计步骤重要性，然后将多余的长度视为惩罚质量，重新分配以更严厉地惩罚低重要性步骤，同时保留高重要性推理。我们使用统一的结果-过程优势在组内相对策略优化中进行优化。广泛的实验表明，SWAP在平均上将推理长度减少64.3%，同时相对于基础模型将准确性提高5.7%。
英文摘要：
Large reasoning models improve with more test-time computation, but often overthink, producing unnecessarily long chains-of-thought that raise cost without improving accuracy. Prior reinforcement learning approaches typically rely on a single outcome reward with trajectory-level length penalties, which cannot distinguish essential from redundant reasoning steps and therefore yield blunt compression. Although recent work incorporates step-level signals, such as offline pruning, supervised data construction, or verifier-based intermediate rewards, reasoning length is rarely treated as an explicit step-level optimization objective during RL. We propose Step-wise Adaptive Penalization (SWAP), a fine-grained framework that allocates length reduction across steps based on intrinsic contribution. We estimate step importance from the model's on-policy log-probability improvement toward the correct answer, then treat excess length as a penalty mass redistributed to penalize low-importance steps more heavily while preserving high-importance reasoning. We optimize with a unified outcome-process advantage within group-relative policy optimization. Extensive experiments demonstrate that SWAP reduces reasoning length by 64.3% on average while improving accuracy by 5.7% relative to the base model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
