TSPO:突破多轮搜索策略优化中的双重均质化困境
TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization
AI总结:
本文提出TSPO方法,通过引入首次出现潜在奖励机制,解决多轮搜索策略优化中的过程均质化和组内均质化问题,提升模型性能。
摘要:
多轮工具集成推理使大型语言模型能够通过迭代信息检索解决复杂任务。然而,当前用于增强推理的强化学习框架主要依赖稀疏的层次级奖励,导致所谓的'双重均质化困境'。这表现为(1)过程均质化,即生成过程中涉及的思考、推理和工具被忽视;(2)组内均质化,粗粒度的奖励往往导致使用如组相对策略优化(GRPO)等方法时组内优势估计效率低下。为了解决这一问题,我们提出了Turn-level Stage-aware Policy Optimization(TSPO)。TSPO引入了首次出现潜在奖励(FOLR)机制,将部分奖励分配给地面真实答案首次出现的步骤,从而保留过程级信号并增加组内奖励方差,而无需外部奖励模型或任何注释。广泛的实验表明,TSPO显著优于最先进的基线,分别在Qwen2.5-3B和7B模型上实现了平均性能提升24%和13.6%。代码可在https://github.com/Flipped-May/TSPO获取。
英文摘要:
Multi-turn tool-integrated reasoning enables Large Language Models (LLMs) to solve complex tasks through iterative information retrieval. However, current reinforcement learning (RL) frameworks for search-augmented reasoning predominantly rely on sparse outcome-level rewards, leading to a "Double Homogenization Dilemma." This manifests as (1) Process homogenization, where the thinking, reasoning, and tooling involved in generation are ignored. (2) Intra-group homogenization, coarse-grained outcome rewards often lead to inefficiencies in intra-group advantage estimation with methods like Group Relative Policy Optimization (GRPO) during sampling. To address this, we propose Turn-level Stage-aware Policy Optimization (TSPO). TSPO introduces the First-Occurrence Latent Reward (FOLR) mechanism, allocating partial rewards to the step where the ground-truth answer first appears, thereby preserving process-level signals and increasing reward variance within groups without requiring external reward models or any annotations. Extensive experiments demonstrate that TSPO significantly outperforms state-of-the-art baselines, achieving average performance gains of 24% and 13.6% on Qwen2.5-3B and 7B models, respectively. Code is available at https://github.com/Flipped-May/TSPO.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。