KnowGPO：面向搜索智能体的知识分组策略优化
KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents
AI总结：
针对深度搜索智能体决策信用分配难题，提出 KnowGPO 框架，通过知识分组分配过程奖励，在 Xbench 等数据集上较基线提升 3.8-4.2 个百分点，8B 模型性能媲美多数 32B 开源系统。
摘要：
深度搜索智能体通过长 horizon 的多轮交互解决知识密集型任务，但为单个搜索决策分配信用本质上十分困难。基于结果的强化学习仅提供整个轨迹共享的单个终端奖励。近期的过程监督方法旨在以更细粒度分配信用，但它们通常将信用锚定到表面级轨迹状态或模型内部信号，在深度搜索中，由于观测结果异构、检索上下文冗长且存在噪声，这两种方式均变得不可靠。我们提出 KnowGPO，一种强化学习框架，该框架按智能体获取的与任务相关的知识而非表面观测结果对齐轨迹。对于每个查询，KnowGPO 将推导答案所需的信息分解为原子知识条件，并通过轨迹每一步满足的条件子集表示轨迹状态。将具有相同知识状态的轨迹分组以进行组内比较，连续步骤之间的增量知识增益作为基于环境的过程奖励。在 Xbench、BrowseComp-ZH 和 GAIA 上的实验表明，KnowGPO 始终比仅基于结果的基线（GRPO）和过程监督基线（GiGPO、IGPO）高出 3.8 至 4.2 个百分点，我们的 8B 模型可与大多数 32B 开源系统相媲美或超越它们。
英文摘要：
Deep search agents solve knowledge-intensive tasks through long-horizon, multi-turn interactions, where assigning credit to individual search decisions is fundamentally difficult. Outcome-based reinforcement learning provides only a single terminal reward shared across an entire trajectory. Recent process-supervision methods aim to assign credit at finer granularity, but they typically anchor credit to either surface-level trajectory states or model-internal signals, both of which become unreliable in deep search where observations are heterogeneous and retrieved contexts are long and noisy. We propose KnowGPO, a reinforcement learning framework that aligns trajectories by the task-relevant knowledge they have acquired rather than by their surface observations. For each query, KnowGPO decomposes the information required to derive the answer into atomic knowledge conditions, and represents the state of a trajectory at each step by the subset of conditions it has satisfied. Trajectories sharing the same knowledge state are grouped for within-group comparison, and the incremental knowledge gain between consecutive steps serves as an environment-grounded process reward. Experiments on Xbench, BrowseComp-ZH, and GAIA show KnowGPO consistently outperforms outcome-only (GRPO) and process-supervision baselines (GiGPO, IGPO) by 3.8 to 4.2 points, with our 8B model rivaling or exceeding most 32B open-source systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
