arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.00499cs.LGcs.AIstat.ML
收藏

ESPO:熵重要性采样策略优化

ESPO: Entropy Importance Sampling Policy Optimization

LLM Team, Shopee Pte. Ltd.(Shopee 股份有限公司大语言模型团队)

Yuepeng Sheng, Yuwei Huang, Shuman Liu, Anxiang Zeng, Haibo Zhang

AI总结:

ESPO通过结合细粒度更新与稳定训练,解决梯度利用率不足问题,提升大语言模型在复杂推理任务中的性能。

摘要:

强化学习(RL)已成为大型语言模型(LLMs)训练后的核心组成部分,特别是在需要长时间生成 horizon 的复杂推理任务中。然而,实现大规模性能往往引入了训练稳定性与训练效率之间的根本权衡。基于令牌的优化在单个单元上应用细粒度更新,但容易导致梯度估计的高方差,从而导致不稳定的训练动态。相比之下,基于序列的优化通常依赖于激进的裁剪机制来确保稳定的更新。然而,这种设计可能会丢弃大量有效的训练样本,导致梯度利用效率低下和训练效率下降。我们称这种现象为梯度利用率不足。在本文中,我们提出了熵重要性采样策略优化(ESPO),一种新的框架,旨在结合细粒度更新与稳定的训练。ESPO 根据预测熵将序列分解为组,从而实现(1)熵分组重要性采样以捕捉序列内异质性,以及(2)熵自适应裁剪以根据模型不确定性动态分配信任区域。在数学推理基准上的广泛实验表明,ESPO 不仅加快了收敛速度,还实现了最先进的性能,特别是在具有挑战性的数学基准上显著提高了准确性。

英文摘要:

Reinforcement learning (RL) has become a central component of post-training for large language models (LLMs), particularly for complex reasoning tasks that require stable optimization over long generation horizons. However, achieving performance at scale often introduces a fundamental trade-off between training stability and training efficiency. Token-level optimization applies fine-grained updates at the individual units, but is prone to high variance in gradient estimation, which can result in unstable training dynamics. In contrast, Sequence-level optimization often relies on aggressive clipping mechanisms to ensure stable updates. However, such design may discard a large fraction of valid training samples, leading to inefficient gradient utilization and reduced training efficiency. We refer to this phenomenon as gradient underutilization. In this work, we propose Entropy Importance Sampling Policy Optimization (ESPO), a novel framework that aims to combine fine-grained updates with stable training. ESPO decomposes sequences into groups based on predictive entropy, enabling (1) Entropy Grouping Importance Sampling to capture intra-sequence heterogeneity, and (2) Entropy Adaptive Clipping to dynamically allocate trust regions based on model uncertainty. Extensive experiments on mathematical reasoning benchmarks demonstrate that ESPO not only accelerates convergence but also achieves state-of-the-art performance, notably improving accuracy on the challenging mathematical benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑