思考密集，而非冗长：动态解耦条件优势用于高效推理
Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning
AI总结：
DDCA通过动态解耦条件优势提升强化学习的效率-准确性平衡，减少生成token数量并保持准确性。
摘要：
可验证奖励的强化学习（RLVR）可以激发强大的多步推理，但往往鼓励过于冗长的轨迹。此外，组相对优化中的朴素长度惩罚会严重损害准确性。我们归因于两个结构性问题：（i）长度基线稀释，其中错误响应（零长度奖励）稀释组基线并过度惩罚正确解；（ii）难度-惩罚不匹配，静态惩罚无法适应问题难度，在困难实例上抑制必要的推理而在简单实例上留下冗余。我们提出了动态解耦条件优势（DDCA）以分离效率优化与正确性。DDCA在正确响应簇内有条件地计算长度优势以消除基线稀释，并动态调整惩罚强度使用组通过率作为难度的代理。在GSM8K、MATH500、AMC23和AIME25上的实验表明，DDCA相对于自适应基线在效率-准确性权衡上持续改进，减少生成的token数量在较简单任务（如GSM8K）上减少约60%，而在更困难的基准（如AIME25）上减少超过20%，从而保持或提高准确性。代码可在https://github.com/alphadl/DDCA上获得。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) can elicit strong multi-step reasoning, yet it often encourages overly verbose traces. Moreover, naive length penalties in group-relative optimization can severely hurt accuracy. We attribute this failure to two structural issues: (i) Dilution of Length Baseline, where incorrect responses (with zero length reward) depress the group baseline and over-penalize correct solutions; and (ii) Difficulty-Penalty Mismatch, where a static penalty cannot adapt to problem difficulty, suppressing necessary reasoning on hard instances while leaving redundancy on easy ones. We propose Dynamic Decoupled Conditional Advantage (DDCA) to decouple efficiency optimization from correctness. DDCA computes length advantages conditionally within the correct-response cluster to eliminate baseline dilution, and dynamically scales the penalty strength using the group pass rate as a proxy for difficulty. Experiments on GSM8K, MATH500, AMC23, and AIME25 show that DDCA consistently improves the efficiency--accuracy trade-off relative to adaptive baselines, reducing generated tokens by approximately 60% on simpler tasks (e.g., GSM8K) versus over 20% on harder benchmarks (e.g., AIME25), thereby maintaining or improving accuracy. Code is available at https://github.com/alphadl/DDCA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
