arXiv 2510.093692025-10-10cs.CL

Token-Level Policy Optimization: 通过马尔可夫似然将组级奖励与词元级聚合相连接

Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood

Baidu Inc（百度公司）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University（吉林大学教育部符号计算与知识工程重点实验室）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全国家重点实验室）

Xingyu Lin, Yilin Wen, En Wang, Du Su, Wenbin Liu, Chenfu Bao, Zhonghou Lv

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对GRPO在思维链中词元奖励稀疏导致的熵坍缩问题，提出TEPO框架，利用马尔可夫似然将组级奖励与词元级聚合连接，在数学推理任务上取得新最优并提升训练稳定性。

摘要：

组相对策略优化（GRPO）显著提升了大语言模型（LLMs）的推理能力，尤其是数学性能。然而，GRPO及相关熵正则化方法仍面临思维链（CoT）中词元奖励稀疏带来的挑战。现有方法常依赖无差别的词元级熵调整，这往往导致熵坍缩或模型坍缩。在本工作中，我们提出TEPO，一种新颖的词元级框架，通过词元级聚合引入马尔可夫似然（序列似然）将组级奖励与词元相连接。实验表明，TEPO在关键指标（包括@k和准确率）上持续优于现有基线。它不仅为数学推理任务树立了新最先进水平，还显著增强了训练稳定性。

英文摘要：

Group Relative Policy Optimization (GRPO) has significantly advanced the reasoning ability of large language models (LLMs), particularly by boosting their mathematical performance. However, GRPO and related entropy-regularization methods still face challenges rooted in the sparse token rewards inherent to chain-of-thought (CoT). Current approaches often rely on undifferentiated token-level entropy adjustments, which frequently lead to entropy collapse or model collapse. In this work, we propose TEPO, a novel token-level framework that incorporates Markov Likelihood (sequence likelihood) links group-level rewards with tokens via token-level aggregation. Experiments show that TEPO consistently outperforms existing baselines across key metrics (including @k and accuracy). It not only sets a new state of the art on mathematical reasoning tasks but also significantly enhances training stability.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。