通过多智能体强化学习实现链式思维的自我压缩
Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning
AI总结：
通过多智能体强化学习实现链式思维的自我压缩，有效减少响应长度并提升准确性。
摘要：
由冗余推理引起的推理开销会损害交互体验，并严重限制大推理模型的部署。现有的基于强化学习（RL）的解决方案通过将长度惩罚与基于结果的奖励相结合来解决这个问题。这种简单的奖励加权难以在简洁性和准确性之间取得平衡，因为强制简洁可能会损害关键的推理逻辑。在本工作中，我们通过提出一个多智能体RL框架来解决这一限制，该框架有选择地惩罚冗余块，同时保留必要的推理逻辑。我们的框架，称为SCMA（通过MARL的自我压缩），通过两个专门的智能体实例化冗余检测和评估：一个用于将推理过程分解为逻辑块的分段智能体，以及一个用于量化每个块重要性的评分智能体。分段和评分智能体在训练期间共同定义了一个重要加权长度惩罚，激励推理智能体在部署时优先考虑必要的逻辑而不引入推理开销。在不同模型规模上的实证评估显示，SCMA将响应长度减少了11.1%至39.0%，同时将准确性提高了4.33%至10.02%。此外，消融研究和定性分析验证了MARL框架内的协同优化促进了新兴行为，产生了比普通RL范式更强大的LRMs。
英文摘要：
The inference overhead induced by redundant reasoning undermines the interactive experience and severely bottlenecks the deployment of Large Reasoning Models. Existing reinforcement learning (RL)-based solutions tackle this problem by coupling a length penalty with outcome-based rewards. This simplistic reward weighting struggles to reconcile brevity with accuracy, as enforcing brevity may compromise critical reasoning logic. In this work, we address this limitation by proposing a multi-agent RL framework that selectively penalizes redundant chunks, while preserving essential reasoning logic. Our framework, Self-Compression via MARL (SCMA), instantiates redundancy detection and evaluation through two specialized agents: \textbf{a Segmentation Agent} for decomposing the reasoning process into logical chunks, and \textbf{a Scoring Agent} for quantifying the significance of each chunk. The Segmentation and Scoring agents collaboratively define an importance-weighted length penalty during training, incentivizing \textbf{a Reasoning Agent} to prioritize essential logic without introducing inference overhead during deployment. Empirical evaluations across model scales demonstrate that SCMA reduces response length by 11.1\% to 39.0\% while boosting accuracy by 4.33\% to 10.02\%. Furthermore, ablation studies and qualitative analysis validate that the synergistic optimization within the MARL framework fosters emergent behaviors, yielding more powerful LRMs compared to vanilla RL paradigms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
