停止不必要的反思:通过自适应反思和长度协调惩罚训练高效的推理模型
Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty
AI总结:
通过自适应反思和长度协调惩罚方法,提升推理效率和准确性,减少响应长度并提高性能。
摘要:
大型推理模型(LRMs)通过在测试时进行扩展,在复杂的推理任务上展现了显著的性能。然而,它们经常生成过长的思考链,这些链被大量反思驱动,如重复的自我提问和循环推理,导致高令牌消耗、显著的计算开销和增加的延迟,而没有提高准确性,特别是在较小的模型中。我们的观察表明,增加问题的复杂性会引发更多的过度和不必要的反思,这反过来会降低准确性并增加令牌开销。为了解决这一挑战,我们提出了自适应反思和长度协调惩罚(ARLCP),一种新的强化学习框架,旨在动态平衡推理效率和解决方案的准确性。ARLCP引入了两个关键创新:(1)一个反思惩罚,能够自适应地削减不必要的反思步骤,同时保留必要的推理;(2)一个长度惩罚,经过校准以估计问题的复杂性。通过协调这些惩罚,ARLCP鼓励模型生成更简洁和有效的推理路径。我们在五个数学推理基准上使用DeepSeek-R1-Distill-Qwen-1.5B和DeepSeek-R1-Distill-Qwen-7B模型评估了我们的方法。实验结果表明,与现有方法相比,ARLCP在效率-准确性权衡上表现更优。对于1.5B模型,它将平均响应长度减少了53.1%,同时提高了5.8%的准确性。对于7B模型,它实现了35.0%的长度减少,同时提高了2.7%的准确性。代码已发布在https://github.com/ZeweiYu1/ARLCP。
英文摘要:
Large Reasoning Models (LRMs) have demonstrated remarkable performance on complex reasoning tasks by employing test-time scaling. However, they often generate over-long chains-of-thought that, driven by substantial reflections such as repetitive self-questioning and circular reasoning, lead to high token consumption, substantial computational overhead, and increased latency without improving accuracy, particularly in smaller models. Our observation reveals that increasing problem complexity induces more excessive and unnecessary reflection, which in turn reduces accuracy and increases token overhead. To address this challenge, we propose Adaptive Reflection and Length Coordinated Penalty (ARLCP), a novel reinforcement learning framework designed to dynamically balance reasoning efficiency and solution accuracy. ARLCP introduces two key innovations: (1) a reflection penalty that adaptively curtails unnecessary reflective steps while preserving essential reasoning, and (2) a length penalty calibrated to the estimated complexity of the problem. By coordinating these penalties, ARLCP encourages the model to generate more concise and effective reasoning paths. We evaluate our method on five mathematical reasoning benchmarks using DeepSeek-R1-Distill-Qwen-1.5B and DeepSeek-R1-Distill-Qwen-7B models. Experimental results show that ARLCP achieves a superior efficiency-accuracy trade-off compared to existing approaches. For the 1.5B model, it reduces the average response length by 53.1% while simultaneously improving accuracy by 5.8%. For the 7B model, it achieves a 35.0% reduction in length with a 2.7% accuracy gain. The code is released at https://github.com/ZeweiYu1/ARLCP .
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。