通过强化学习增强思维链，重新思考大语言模型中的推理质量
Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
AI总结：
提出DRER奖励框架，通过推理质量奖励和动态长度优势优化思维链，在Logictree基准上7B模型达GPT-o3-mini水平，置信度提升30%。
摘要：
强化学习（RL）近来已成为增强大语言模型（LLMs）推理能力的主导范式。然而，常用于数学或编程基准测试的基于规则的奖励函数仅评估答案格式和正确性，无法提供关于所诱导的思维链（CoT）是否真正改善答案的信号。此外，此类特定任务的训练对逻辑深度的控制有限，因此可能无法揭示模型的真实推理能力。我们提出动态推理效率奖励（DRER）——一个即插即用的RL奖励框架，重塑奖励和优势信号。（i）推理质量奖励为那些可证明提高正确答案可能性的推理链分配细粒度信用，直接激励具有有益CoT标记的轨迹。（ii）动态长度优势衰减那些长度偏离验证导出阈值的响应的优势，从而稳定训练。为促进严格评估，我们还发布了Logictree，一个动态构建的演绎推理数据集，既作为RL训练数据，又作为综合基准。实验证实了DRER的有效性：我们的7B模型在Logictree上经过400步训练达到GPT-o3-mini级别性能，而CoT增强答案的平均置信度提高了30%。该模型还在多种逻辑推理数据集以及数学基准AIME24上展现出泛化能力。这些结果阐明了RL如何塑造CoT行为，并为增强大语言模型中的形式推理技能绘制了一条实用路径。所有代码和数据可在仓库https://github.com/Henryhe09/DRER中获取。
英文摘要：
Reinforcement learning (RL) has recently become the dominant paradigm for strengthening the reasoning abilities of large language models (LLMs). Yet the rule-based reward functions commonly used on mathematical or programming benchmarks assess only answer format and correctness, providing no signal as to whether the induced Chain-of-Thought (CoT) actually improves the answer. Furthermore, such task-specific training offers limited control over logical depth and therefore may fail to reveal a model's genuine reasoning capacity. We propose Dynamic Reasoning Efficiency Reward (DRER) -- a plug-and-play RL reward framework that reshapes both reward and advantage signals. (i) A Reasoning Quality Reward assigns fine-grained credit to those reasoning chains that demonstrably raise the likelihood of the correct answer, directly incentivising the trajectories with beneficial CoT tokens. (ii) A Dynamic Length Advantage decays the advantage of responses whose length deviates from a validation-derived threshold, stabilising training. To facilitate rigorous assessment, we also release Logictree, a dynamically constructed deductive reasoning dataset that functions both as RL training data and as a comprehensive benchmark. Experiments confirm the effectiveness of DRER: our 7B model attains GPT-o3-mini level performance on Logictree with 400 trianing steps, while the average confidence of CoT-augmented answers rises by 30%. The model further exhibits generalisation across diverse logical-reasoning datasets, and the mathematical benchmark AIME24. These results illuminate how RL shapes CoT behaviour and chart a practical path toward enhancing formal-reasoning skills in large language models. All code and data are available in repository https://github.com/Henryhe09/DRER.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
