arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.14386cs.CL
收藏

超越令牌级策略梯度:用于大语言模型复杂推理的多令牌策略梯度优化

Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models

School of Computer Science and Technology, Harbin Institute of Technology, China(哈尔滨工业大学计算机科学与技术学院);Baidu Inc.(百度公司)

Mufan Xu, Kehai Chen, Xuefeng Bai, Zhengyu Niu, Muyun Yang, Tiejun Zhao, Min Zhang

AI总结:

本文提出MPO框架,通过多令牌级优化提升大语言模型在复杂推理任务中的性能。

摘要:

现有用于自回归语言模型的策略梯度方法通常将后续令牌逐一作为策略中的动作进行选择。尽管在许多生成任务中有效,但这种方法可能无法完全捕捉复杂推理任务的结构,其中单一语义决策往往需要多个令牌来实现——例如在定义变量或组合方程时。这引入了令牌级优化与这些场景中固有块级性质之间的潜在不匹配。为弥合这一差距,我们提出了多令牌策略梯度优化(MPO),一种将连续K个令牌序列视为统一语义动作的框架。这种块级视角使我们的方法能够捕捉推理轨迹的组合结构,并支持对一致、更高层次目标的优化。在数学推理和编码基准测试中,MPO在数学推理和编码基准测试中表现优于标准令牌级策略梯度基线,突显了令牌级策略梯度在复杂推理中的局限性,推动未来研究超越令牌级粒度以应对推理密集型语言任务。

英文摘要:

Existing policy-gradient methods for auto-regressive language models typically select subsequent tokens one at a time as actions in the policy. While effective for many generation tasks, such an approach may not fully capture the structure of complex reasoning tasks, where a single semantic decision is often realized across multiple tokens--for example, when defining variables or composing equations. This introduces a potential mismatch between token-level optimization and the inherently block-level nature of reasoning in these settings. To bridge this gap, we propose Multi-token Policy Gradient Optimization (MPO), a framework that treats sequences of K consecutive tokens as unified semantic actions. This block-level perspective enables our method to capture the compositional structure of reasoning trajectories and supports optimization over coherent, higher-level objectives. Experiments on mathematical reasoning and coding benchmarks show that MPO outperforms standard token-level policy gradient baselines, highlight the limitations of token-level policy gradients for complex reasoning, motivating future research to look beyond token-level granularity for reasoning-intensive language tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑