TRiMS:通过强化学习实现高效的实时最小充分长度推理跟踪
TRiMS: Real-Time Tracking of Minimal Sufficient Length for Efficient Reasoning via RL
AI总结:
本文提出TRiMS,通过强化学习和最小充分长度指标,实现高效推理压缩,减少80%以上的推理token数量并提升准确性。
摘要:
大型语言模型通过长链式思维序列在复杂推理中取得突破,但往往导致严重的推理膨胀,造成大量计算冗余。为最大化每token的智能,我们引入了一个理论度量指标MSL-最小充分长度。MSL严格刻画了保持答案正确性的最短推理长度。我们基于独立采样序列提供递归定义,并证明其极限的存在,建立了推理链压缩的第一个可测量下界。基于主流CoT压缩策略的分析,我们识别出使模型接近MSL的关键结构性因素。基于这些见解,我们提出TRiMS,其在训练过程中结合GRPO算法和基于MSL的估计,通过动态批量聚合和利用批量级标准差的优勢计算来缓解训练过程中的不稳定性。TRiMS在所有基准上实现了超过80%的CoT token减少,并带来了小幅的准确性提升。
英文摘要:
Large language models achieve breakthroughs in complex reasoning via long chain-of-thought sequences. However, this often leads to severe reasoning inflation, causing substantial computational redundancy. To maximize Intelligence per Token, we introduce a theoretical metric, MSL-Minimal Sufficient Length. MSL rigorously characterizes the shortest reasoning length that preserves answer correctness. We provide a recursive definition based on independently sampled sequences and prove the existence of its limit, establishing the first measurable lower bound for reasoning-chain compression. Building on an analysis of mainstream CoT compression strategies, we identify key structural factors enabling a model to approach MSL. Based on these insights, we propose TRiMS which employs the GRPO algorithm in conjunction with MSL-based estimation during training, while mitigating instabilities during the training process through dynamic batch aggregation and advantage computation using batch-level standard deviation. TRiMS achieves over 80% CoT token reduction with a minor accuracy boost across all benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。