SmartThinker: 渐进式思维链长度校准以实现高效的大语言模型推理
SmartThinker: Progressive Chain-of-Thought Length Calibration for Efficient Large Language Model Reasoning
AI总结：
针对大型推理模型输出冗余问题，提出基于GRPO的渐进式CoT长度校准方法SmartThinker，通过动态估计最优长度和调节长度奖励系数，在压缩响应长度同时提升准确率。
摘要：
大型推理模型（LRMs），如OpenAI o1和DeepSeek-R1，通过采用长思维链（CoT）推理路径在复杂任务上实现了高准确率。然而，这些过程固有的冗长常常导致冗余和过度思考。为了解决这一问题，现有工作利用组相对策略优化（GRPO）来减少LRM的输出长度，但其静态长度奖励设计无法根据问题相对难度和响应长度分布动态调整，导致过度压缩和准确率下降。因此，我们提出SmartThinker，一种新颖的基于GRPO的高效推理方法，具有渐进式CoT长度校准。SmartThinker有两个贡献：首先，它在训练期间动态估计具有峰值准确率的最优长度，并引导过长响应朝向该长度，以减少响应长度同时保持准确率。其次，它动态调节长度奖励系数，以避免对正确推理路径的不当惩罚。大量实验结果表明，SmartThinker在提高准确率的同时实现了高达52.5%的平均长度压缩，并在AIME25等具有挑战性的基准上实现了高达16.6%的准确率提升。源代码可在https://github.com/SJTU-RTEAS/SmartThinker获取。
英文摘要：
Large reasoning models (LRMs) like OpenAI o1 and DeepSeek-R1 achieve high accuracy on complex tasks by adopting long chain-of-thought (CoT) reasoning paths. However, the inherent verbosity of these processes frequently results in redundancy and overthinking. To address this issue, existing works leverage Group Relative Policy Optimization (GRPO) to reduce LRM output length, but their static length reward design cannot dynamically adapt according to the relative problem difficulty and response length distribution, causing over-compression and compromised accuracy. Therefore, we propose SmartThinker, a novel GRPO-based efficient reasoning method with progressive CoT length calibration. SmartThinker makes a two-fold contribution: First, it dynamically estimates the optimal length with peak accuracy during training and guides overlong responses toward it to reduce response length while sustaining accuracy. Second, it dynamically modulates the length reward coefficient to avoid the unwarranted penalization of correct reasoning paths. Extensive experiment results show that SmartThinker achieves up to 52.5% average length compression with improved accuracy, and achieves up to 16.6% accuracy improvement on challenging benchmarks like AIME25. The source code can be found at https://github.com/SJTU-RTEAS/SmartThinker.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
