ConciseRL：面向高效推理模型的简洁性引导强化学习
ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models
AI总结：
本文提出ConciseRL，利用大语言模型评判的简洁性分数作为强化学习奖励，引导模型生成正确且简洁的推理轨迹，在MATH和TheoremQA上显著减少令牌使用并提升准确性。
摘要：
大型语言模型通过将复杂问题分解为结构化的推理步骤而表现出色。然而，推理轨迹往往超出得出正确答案所需的范围，导致计算浪费、可读性降低和幻觉产生。为了解决这一问题，我们引入了一种新颖的、无需超参数的简洁性分数，在强化学习框架中用作奖励信号，以引导模型生成正确且简洁的推理轨迹。该分数由作为评判者的大型语言模型进行评估，从而能够提供超越简单令牌长度的动态、上下文感知反馈。我们的方法在MATH数据集上实现了最先进的效率-准确性权衡，在简单问题上将令牌使用量减少高达31倍，同时将准确性提高7%；在最难的问题上，它以高达3.6倍的令牌减少量，在准确性上比完整推理高出7.5%。在TheoremQA上，我们的方法以12.5倍的令牌减少量将准确性提高了2.2%。我们还对评判模型、奖励组成和问题难度进行了消融研究，结果表明我们的方法根据问题难度动态调整推理长度，并显著受益于更强的评判者。代码、模型权重和数据集已在https://github.com/RazvanDu/ConciseRL开源。
英文摘要：
Large language models excel at complex tasks by breaking down problems into structured reasoning steps. However, reasoning traces often extend beyond reaching a correct answer, causing wasted computation, reduced readability, and hallucinations. To address this, we introduce a novel hyperparameter-free conciseness score used as a reward signal within a reinforcement learning framework to guide models toward generating correct and concise reasoning traces. This score is evaluated by a large language model acting as a judge, enabling dynamic, context-aware feedback beyond simple token length. Our method achieves state-of-the-art efficiency-accuracy trade-offs on the MATH dataset, reducing token usage by up to 31x on simple problems while improving accuracy by 7%, and on the hardest problems, it outperforms full reasoning by +7.5% accuracy with up to 3.6x fewer tokens. On TheoremQA, our method improves accuracy by +2.2% using 12.5x fewer tokens. We also conduct ablation studies on the judge model, reward composition, and problem difficulty, showing that our method dynamically adapts reasoning length based on problem difficulty and benefits significantly from stronger judges. The code, model weights, and datasets are open-sourced at https://github.com/RazvanDu/ConciseRL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
