arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.10048cs.LGcs.AI
收藏

通过细粒度分组策略优化实现长链式思考压缩

Long Chain-of-Thought Compression via Fine-Grained Group Policy Optimization

Xinchen Han, Hossam Afifi, Michel Marot, Xilu Wang, Lu Yin

AI总结:

本文提出细粒度分组策略优化算法,通过优化分组响应的权重实现高效的链式思考压缩,同时解决GRPO的两个主要问题。

摘要:

大型语言模型(LLMs)常常生成不必要的冗长链式思考(CoT)推理,这会增加计算成本和延迟,而没有带来相应的性能提升。在本文中,我们提出了细粒度分组策略优化(FGO),这是一种强化学习(RL)算法,通过将响应细分为小组并根据长度和熵分配适当的权重,从而实现有效的CoT压缩。同时,作为分组相对策略优化(GRPO)的增强变种,FGO成功解决了GRPO的两个主要局限性:数据利用效率低下和熵崩溃。我们评估了FGO在多个推理LLMs和基准测试上,包括MATH500、AIME24、AMC23和Minerva。实验结果表明,FGO在不降低性能的情况下实现了高效的CoT压缩,同时解决了GRPO的关键局限性。代码:https://github.com/Mr-XcHan/FGO。

英文摘要:

Large Language Models (LLMs) often generate unnecessarily verbose Chain-of-Thought (CoT) reasoning that increases computational costs and latency without proportional performance gains. In this paper, we propose Fine-grained Group policy Optimization (FGO), a Reinforcement Learning (RL) algorithm that refines group responses by subdividing them and assigning appropriate weights based on length and entropy, thereby enabling effective CoT compression. Meanwhile, as an enhanced variant of Group Relative Policy Optimization (GRPO), FGO successfully addresses two major limitations of the GRPO: inefficient data utilization and entropy collapse. We evaluate FGO on multiple reasoning LLMs and benchmarks, including MATH500, AIME24, AMC23, and Minerva. Experimental results show that FGO achieves efficient CoT compression without degrading performance, and simultaneously resolves the key limitations of GRPO. Code: https://github.com/Mr-XcHan/FGO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑