开篇即胜，半途而废：基于前缀优化的强化学习用于大语言模型推理
Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning
AI总结：
PPPO通过优化LLM推理的前缀部分，提升推理能力，实验显示其在推理任务中表现更优。
摘要：
可验证奖励的强化学习（RLVR）显著增强了大型语言模型（LLMs）的推理能力。当前的RLVR方法通常在所有生成的标记上进行训练，但忽略了哪些标记（例如前缀标记）实际上有助于推理。这种统一的训练策略在优化低回报标记上耗费大量精力，从而阻碍了高回报标记的潜在改进，并降低了整体训练效果。为了解决这个问题，我们提出了一种新的RLVR方法，称为逐步前缀标记策略优化（PPPO），它强调了生成输出中前缀部分的重要性。具体来说，受已建立的人类思维理论路径依赖性启发，其中早期想法显著限制了后续思维轨迹，我们识别出在LLM推理中类似的现象，称为开始锁定效应（BLE）。PPPO利用这一发现，通过专注于LLM的前缀推理过程来聚焦其优化目标。这种针对性的优化策略可以积极影响后续的推理过程，最终提高最终结果。为了提高LLM对如何以高质量开始推理的学习效果，PPPO引入了两种训练策略：（a）逐步前缀保留，通过在训练过程中增加保留前缀标记的比例来形成逐步学习过程；（b）延续累积奖励，通过为一个前缀标记序列采样多个延续并累积它们的分数作为奖励信号来减轻奖励偏差。在各种推理任务上的广泛实验结果表明，我们提出的方法PPPO在仅26.17%的训练标记上实现了18.02%的准确率提升，优于代表性的RLVR方法。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capability of Large Language Models (LLMs). Current RLVR approaches typically conduct training across all generated tokens, but neglect to explore which tokens (e.g., prefix tokens) actually contribute to reasoning. This uniform training strategy spends substantial effort on optimizing low-return tokens, which in turn impedes the potential improvement from high-return tokens and reduces overall training effectiveness. To address this issue, we propose a novel RLVR approach called Progressive Prefix-token Policy Optimization (PPPO), which highlights the significance of the prefix segment of generated outputs. Specifically, inspired by the well-established human thinking theory of Path Dependence, where early-stage thoughts substantially constrain subsequent thinking trajectory, we identify an analogous phenomenon in LLM reasoning termed Beginning Lock-in Effect (BLE). PPPO leverages this finding by focusing its optimization objective on the prefix reasoning process of LLMs. This targeted optimization strategy can positively influence subsequent reasoning processes, and ultimately improve final results. To improve the learning effectiveness of LLMs on how to start reasoning with high quality, PPPO introduces two training strategies: (a) Progressive Prefix Retention, which shapes a progressive learning process by increasing the proportion of retained prefix tokens during training; (b) Continuation Accumulated Reward, which mitigates reward bias by sampling multiple continuations for one prefix token sequence, and accumulating their scores as the reward signal. Extensive experimental results on various reasoning tasks demonstrate that our proposed PPPO outperforms representative RLVR methods, with the accuracy improvements of 18.02% on only 26.17% training tokens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
