arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.14967cs.CLcs.AIcs.LG
收藏

基于信息增益的策略优化:一种用于多轮搜索代理的简单而有效的方法

Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents

Venus Team, Ant Group(蚂蚁集团 Venus 团队);Renmin University of China(中国人民大学)

Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, Zhenzhe Ying

AI总结:

本文提出基于信息增益的策略优化框架,通过密集内在监督提升多轮代理训练效果,实验表明其在多轮场景中优于现有方法,准确性和数据效率更高。

摘要:

大型语言模型(LLM)基于的代理越来越多地通过强化学习(RL)训练,以增强其通过工具使用与外部环境交互的能力,特别是在需要多轮推理和知识获取的搜索设置中。然而,现有方法通常依赖于基于结果的奖励,这些奖励仅在生成最终答案时提供。这种奖励稀疏性在多轮设置中尤为突出,长轨迹加剧了三个关键问题:(i)优势崩溃,所有轨迹获得相同奖励,提供无用的学习信号;(ii)缺乏细粒度信用分配,中间步骤的正确性被掩盖,尤其是在长周期任务中;(iii)低效的样本利用率,每个轨迹仅产生一个结果信号,导致数据利用低。本文提出信息增益基于的策略优化(IGPO),一种简单而有效的RL框架,为多轮代理训练提供密集的内在监督。IGPO将每个交互步骤视为获取地面真实信息的渐进过程,并将步骤级奖励定义为策略产生正确答案概率的边际增加。不同于先前的步骤级奖励方法,这些方法依赖于外部奖励模型或昂贵的蒙特卡洛估计,IGPO直接从模型自身的信念更新中推导内在奖励。这些内在步骤级奖励与结果级监督结合,形成密集的奖励信号。在域内和域外基准上的广泛实验表明,IGPO在多轮场景中一致优于强大的基线方法,实现更高的准确性和改进的数据效率。我们的代码可在https://github.com/GuoqingWang1/IGPO上获取。

英文摘要:

Large language model (LLM)-based agents are increasingly trained with reinforcement learning (RL) to enhance their ability to interact with external environments through tool use, particularly in search-based settings that require multi-turn reasoning and knowledge acquisition. However, existing approaches typically rely on outcome-based rewards that are only provided exclusively upon generating the final answer. This reward sparsity becomes particularly problematic in multi-turn settings, where long trajectories exacerbate three critical issues: (i) advantage collapse, where all rollouts receive identical rewards and provide no useful learning signals; (ii) lack of fine-grained credit assignment, where the correctness of intermediate turns is obscured, especially in long-horizon tasks; and (iii) poor sample efficiency, where each rollout yields only a single outcome signal, leading to low data utilization. In this paper, we propose Information Gain-based Policy Optimization (IGPO), a simple yet effective RL framework that provides dense and intrinsic supervision for multi-turn agent training. IGPO models each interaction turn as an incremental process of acquiring information about the ground truth, and defines turn-level rewards as the marginal increase in the policy's probability of producing the correct answer. Unlike prior process-level reward approaches that depend on external reward models or costly Monte Carlo estimation, IGPO derives intrinsic rewards directly from the model's own belief updates. These intrinsic turn-level rewards are combined with outcome-level supervision to form dense reward signals. Extensive experiments on both in-domain and out-of-domain benchmarks demonstrate that IGPO consistently outperforms strong baselines in multi-turn scenarios, achieving higher accuracy and improved data efficiency. Our code is available at https://github.com/GuoqingWang1/IGPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑