GTPO和GRPO-S：基于策略熵的token和序列级奖励塑造
GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy
AI总结：
本文提出GTPO和GRPO-S两种算法，通过引入熵权机制实现token和序列级的细粒度奖励塑造，提升大型语言模型在长链推理任务中的稳定性与性能。
摘要：
强化学习（RL）对于增强大型语言模型（LLM）的推理能力至关重要，然而主流算法如GRPO和DAPO仍然受到粗粒度信用分配范式的限制，其中同一响应中的所有token都获得相同的奖励。在本文中，我们提出了动态熵权，系统地定义基于熵的权重比$\frac{H_{i,t}}{\sum_{k=1}^{n} H_{k,t}}$及其类似变体，通过两种新算法重新分配奖励以获得细粒度奖励：组token策略优化（GTPO），为每个token分配熵加权奖励，并合成token特定的优势函数以驱动模型朝最优路径发展；以及类似的序列级GRPO（GRPO-S），将此设计扩展到序列层面，并在长链式推理（CoT）任务中表现出更稳定的性能。
英文摘要：
Reinforcement Learning (RL) is pivotal for enhancing Large Language Model (LLM) reasoning, yet mainstream algorithms such as GRPO and DAPO remain constrained by a coarse-grained credit assignment paradigm, where all tokens within the same response receive the identical reward. In this paper, we propose Dynamic Entropy Weighting, systematically define entropy-based weight ratios $\frac{H_{i,t}}{\sum_{k=1}^{n} H_{k,t}}$ and similar variants to redistribute rewards and get fine-grained rewards through two new algorithms: Group Token Policy Optimization (GTPO), which assigns an entropy-weighted reward to each token and synthesizes token-specific advantage function to drive the model toward optimal path, and the analogous algorithm Sequence-Level GRPO (GRPO-S), which extends this design to the sequence level and exhibits superior stability in long Chain-of-Thought (CoT) reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
