平衡推理负担：基于长度再分配的难度差异化策略优化用于高效且鲁棒的强化学习
Balancing the Reasoning Load: Difficulty-Differentiated Policy Optimization with Length Redistribution for Efficient and Robust Reinforcement Learning
AI总结：
本文提出DDPO算法，通过区分任务难度优化输出长度，提升强化学习的效率与鲁棒性，实验表明在多个基准上准确率提升1.85%的同时平均回答长度减少12%。
摘要：
大型推理模型（LRMs）虽表现出色，但存在过度思考问题，常生成过长冗余答案。针对超出模型能力的问题，LRMs易表现出过度自信现象，生成过短但错误答案，影响性能。为此，本文提出难度差异化策略优化（DDPO），一种高效的强化学习算法，基于过度自信现象分别优化简单和复杂任务。具体而言，对于简单任务减少输出长度而不影响准确性，对于复杂任务扩展探索空间以提高性能。进一步推导了最大化预期准确性的理论条件，要求长度分布尽可能接近最优长度且集中。基于此，提出使用难度级别平均作为长度优化的基准。在领域内和领域外基准上的广泛实验验证了DDPO的优越性和有效性。相比GRPO，DDPO在多个基准上将平均回答长度减少12%，同时准确率提升1.85%，实现了准确率与长度之间的更好权衡。代码可在https://github.com/Yinan-Xia/DDPO获取。
英文摘要：
Large Reasoning Models (LRMs) have shown exceptional reasoning capabilities, but they also suffer from the issue of overthinking, often generating excessively long and redundant answers. For problems that exceed the model's capabilities, LRMs tend to exhibit the overconfidence phenomenon, generating overly short but incorrect answers, which may contribute to suboptimal performance. To address these issues, we propose Difficulty-Differentiated Policy Optimization (DDPO), an efficient reinforcement learning algorithm that optimizes simple and complex tasks separately based on the overconfidence phenomenon. Specifically, it reduces the output length for simple tasks without compromising accuracy, while for complex tasks, it expands the exploration space to improve performance. We further derive the theoretical conditions for maximizing expected accuracy, which require the length distribution to closely approximate the optimal length and be as concentrated as possible. Based on these conditions, we propose using the difficulty-level average as a well-founded reference for length optimization. Extensive experiments on both in-domain and out-of-domain benchmarks validate the superiority and effectiveness of DDPO. Compared to GRPO, DDPO reduces the average answer length by 12% while improving accuracy by 1.85% across multiple benchmarks, achieving a better trade-off between accuracy and length. The code is available at https://github.com/Yinan-Xia/DDPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
