HMPO: 用于思维链压缩的混合中位数长度策略优化
HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression
AI总结：
提出HMPO，一种单阶段强化学习框架，通过自适应中位数预算、余弦衰减令牌奖励和乘法奖励公式，在数学数据上训练后实现19%-46%的令牌压缩且精度损失极小，并泛化至多种任务。
摘要：
大型语言模型通过扩展的思维链推理取得了显著性能，但这一冗长过程带来了大量推理开销。现有的思维链压缩方法面临不灵活的手动长度预算、计算昂贵的多阶段训练流程以及仅适用于小模型的脆弱可扩展性。我们提出HMPO（混合中位数长度策略优化），一种经济高效的单阶段强化学习框架。HMPO通过三个协同组件高效压缩思维链：基于成功轨迹的自适应中位数预算以消除手动调整、用于平滑长度惩罚的余弦衰减令牌奖励，以及通过严格优先考虑答案正确性来大幅减轻琐碎奖励破解的乘法奖励公式。仅在数学数据上训练，HMPO无缝泛化到数学、代码、科学和指令遵循任务。在从9B到122B参数、涵盖密集和混合专家架构的大规模实验中，HMPO实现了19%-46%的令牌压缩，精度下降可忽略，同时与现有的多阶段基线相比大幅降低了训练成本。
英文摘要：
Large language models achieve remarkable performance via extended chain-of-thought (CoT) reasoning, yet this lengthy process incurs substantial inference overhead. Existing CoT compression methods struggle with inflexible manual length budgets, computationally expensive multi-stage training pipelines, and fragile scalability restricted to small models. We propose HMPO (Hybrid Median-length Policy Optimization), a cost-effective, single-stage reinforcement learning framework. HMPO efficiently compresses CoT via three synergistic components: an adaptive median-based budget derived from successful rollouts to eliminate manual tuning, a cosine-decay token reward for smooth length penalization, and a multiplicative reward formulation that substantially mitigates trivial reward hacking by strictly prioritizing answer correctness. Trained exclusively on mathematical data, HMPO generalizes seamlessly across math, code, science, and instruction-following tasks. Extensive experiments scaling from 9B to 122B parameters across dense and Mixture-of-Experts (MoE) architectures demonstrate that HMPO achieves 19%--46% token compression with negligible accuracy degradation, all while drastically reducing training costs compared to existing multi-stage baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
