arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.22642cs.LG
收藏

压缩简单问题,探索困难问题:面向高效LLM推理的难度感知熵正则化

Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoning

Nanjing University of Aeronautics and Astronautics(南京航空航天大学);College of Computer Science and Technology(计算机科学与技术学院);Didichuxing Co. Ltd(滴滴出行有限公司)

Qin-Wen Luo, Sheng Ren, Xiang Chen, Rui Liu, Jun Fang, Naiqiang Tan, Sheng-Jun Huang

AI总结:

提出CEEH方法,通过难度感知的熵正则化,在强化学习中对简单问题压缩推理长度,对困难问题保持探索空间,从而在减少响应长度的同时维持推理准确性。

摘要:

思维链(CoT)极大地增强了大型语言模型(LLM)处理复杂推理任务的能力,但显式推理步骤的冗长特性导致了高昂的推理延迟和计算成本,限制了实际部署。尽管现有的压缩方法——从自训练到带长度约束的强化学习(RL)——试图缓解这一问题,但它们往往以牺牲推理能力为代价来换取简洁性。我们识别出这些方法中的一个关键失败模式:显式优化更短的轨迹会导致熵的快速崩溃,这过早地缩小了探索空间,并抑制了有效推理路径的发现,尤其是对于需要大量推理的困难问题。为了解决这个问题,我们提出了CEEH(压缩简单问题,探索困难问题),一种面向基于RL的高效推理的难度感知方法。CEEH动态评估实例难度以应用选择性熵正则化:它对当前困难问题保留多样化的搜索空间以确保鲁棒性,同时允许对推理路径已明确的简单实例进行激进压缩。此外,我们引入了一个锚定到历史上最短正确响应的动态最优长度惩罚,这有效抵消了熵引起的长度膨胀并稳定了奖励信号。在六个推理基准测试中,CEEH在保持与基础模型相当的准确性的同时,一致地减少了响应长度,并相对于仅长度优化提高了Pass@k。

英文摘要:

Chain-of-Thought (CoT) has substantially empowered Large Language Models (LLMs) to tackle complex reasoning tasks, yet the verbose nature of explicit reasoning steps incurs prohibitive inference latency and computational costs, limiting real-world deployment. While existing compression methods - ranging from self-training to Reinforcement Learning (RL) with length constraints - attempt to mitigate this, they often sacrifice reasoning capability for brevity. We identify a critical failure mode in these approaches: explicitly optimizing for shorter trajectories triggers rapid entropy collapse, which prematurely shrinks the exploration space and stifles the discovery of valid reasoning paths, particularly for challenging questions requiring extensive deduction. To address this issue, we propose Compress responses for Easy questions and Explore Hard ones (CEEH), a difficulty-aware approach to RL-based efficient reasoning. CEEH dynamically assesses instance difficulty to apply selective entropy regularization: it preserves a diverse search space for currently hard questions to ensure robustness, while permitting aggressive compression on easier instances where the reasoning path is well-established. In addition, we introduce a dynamic optimal-length penalty anchored to the historically shortest correct response, which effectively counteracts entropy-induced length inflation and stabilizes the reward signal. Across six reasoning benchmarks, CEEH consistently reduces response length while maintaining accuracy comparable to the base model, and improves Pass@k relative to length-only optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑