arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.03327cs.LGcs.AI
收藏

DGPO:基于分布的策略优化用于细粒度信用分配

TIDE: Token-Level Credit Assignment via Entropy-Gated Bounded Deviation

Peking University(北京大学);SJTU(上海交通大学);Tsinghua University(清华大学)

Qiaoman Zhang, Shunhua Min, Xiang He, Zhenghan Zhang, Yuting Zhang, Guangxian Li, Yunzhu Kuang, Rui Jiang, Jingyan Zhang, Chaoyue Li, Jiayu Ding

AI总结:

DGPO通过引入分布引导的策略优化框架,解决传统方法在细粒度信用分配中的不足,通过Hellinger距离和熵门机制实现安全探索与有效区分真实推理突破,提升大语言模型在复杂推理任务中的性能。

摘要:

强化学习对于对齐大型语言模型以执行复杂推理任务至关重要。然而,当前算法如组相对策略优化在长链思考生成中面临粗粒度的序列级信用分配问题,严重阻碍了关键推理步骤的隔离。此外,标准无界Kullback-Leibler散度惩罚导致严重的梯度不稳定性及模式寻求保守性,最终抑制了新推理轨迹的发现。为克服这些限制,我们引入了分布引导的策略优化,一种无需批评者的强化学习框架,将分布偏差重新解释为引导信号而非刚性惩罚。DGPO用受限制的Hellinger距离取代易波动的KL散度,以安全量化令牌级探索而无梯度爆炸风险。为有效区分真实推理突破与幻觉噪声,我们提出熵门机制,通过策略的认知不确定性放大此偏差。通过动态重新分配粗粒度序列级优势至个体令牌基于这些门控分数,DGPO强烈激励关键探索步骤,同时抑制不必要、低熵偏差。因此,DGPO完全消除了传统令牌级KL惩罚,实现细粒度信用再分配,而无需额外价值网络的计算开销。广泛的实证评估表明,DGPO在无批评者对齐中达到新状态。值得注意的是,在Qwen2.5-32B架构上,DGPO在具有挑战性的AIME2024和AIME2025基准上分别达到60.0%的Avg@32准确率和46.0%的Avg@32准确率,显著优于竞争基线如DAPO。

英文摘要:

Reinforcement learning (RL) is central for aligning large language models (LLMs) with complex reasoning, yet the leading critic-free algorithm, Group Relative Policy Optimization (GRPO), broadcasts a single sequence-level advantage uniformly across long Chain-of-Thought (CoT) generations and constrains exploration with an unbounded Kullback-Leibler (KL) penalty, which dilutes token-level supervision and induces gradient instability and mode-seeking conservatism. We propose TIDE (Token-level Importance via Deviation and Entropy), a critic-free RL framework that reinterprets distribution deviation as a guiding signal rather than a rigid penalty. TIDE replaces the volatile reverse KL divergence with the bounded Hellinger distance, safely quantifying token-level exploration without risk of gradient explosion. To separate genuine reasoning breakthroughs from confident hallucinations, an entropy gating mechanism scales this deviation by the policy's predictive uncertainty, and the gated scores redistribute the sequence-level advantage across tokens. The resulting updates amplify exploratory steps while suppressing low-entropy deviations, without a value network and with negligible overhead. Across in-domain, held-out, and out-of-domain benchmarks, TIDE outperforms strong critic-free baselines: on the Qwen2.5-32B backbone it reaches 60.0% and 46.0% Avg@32 accuracy on AIME 2024 and AIME 2025, respectively, above competitive baselines such as DAPO. Token-level attribution, causal perturbation, and exploration analyses further support the interpretation that the entropy-gated deviation identifies pivotal reasoning steps rather than merely perturbing optimization dynamics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑