arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.00568cs.CL
收藏

ReSeek:一种具有指导性奖励的自我纠正搜索代理框架

ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards

Basic Algorithm Center, PCG, Tencent, Shenzhen, China(腾讯基本算法中心、PCG、腾讯,深圳,中国);Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China(清华大学深圳国际研究生学院,清华大学,深圳,中国)

Shiyu Li, Yang Tang, Yifan Wang, Peiming Li, Xi Chen

AI总结:

本文提出ReSeek框架,通过引入自我纠正机制和密集指导性奖励函数,提升搜索代理在复杂任务中的表现和路径忠实度。

摘要:

基于大语言模型的搜索代理在解决知识密集型任务中展现出巨大潜力。强化学习(RL)已成为训练这些代理进行复杂多步骤推理的强大范式。然而,先前基于RL的方法往往依赖稀疏或规则奖励,导致代理无法恢复错误的推理路径。为解决这些问题,我们提出了ReSeek,一种新的自我纠正框架。该框架引入了一种自我纠正机制,使代理能够在回合中动态识别并恢复错误的搜索路径。通过调用特殊JUDGE动作,代理可以判断信息并重新规划搜索策略。为了引导此过程,我们设计了一个密集、指导性的过程奖励函数,将其分解为检索事实信息的正确性奖励和找到对查询真正有用的信息的效用奖励。此外,为了减轻现有数据集中的数据污染风险,我们引入了FictionalHot,一个具有最新整理问题的挑战性基准,需要复杂的推理。经过大量实验表明,使用ReSeek训练的代理在任务成功率和路径忠实度上显著优于SOTA基线。

英文摘要:

Search agents powered by Large Language Models (LLMs) have demonstrated significant potential in tackling knowledge-intensive tasks. Reinforcement learning (RL) has emerged as a powerful paradigm for training these agents to perform complex, multi-step reasoning. However, prior RL-based methods often rely on sparse or rule-based rewards, which can lead agents to commit to suboptimal or erroneous reasoning paths without the ability to recover. To address these limitations, we propose ReSeek, a novel self-correcting framework for training search agents. Our framework introduces a self-correction mechanism that empowers the agent to dynamically identify and recover from erroneous search paths during an episode. By invoking a special JUDGE action, the agent can judge the information and re-plan its search strategy. To guide this process, we design a dense, instructive process reward function, which decomposes into a correctness reward for retrieving factual information and a utility reward for finding information genuinely useful for the query. Furthermore, to mitigate the risk of data contamination in existing datasets, we introduce FictionalHot, a new and challenging benchmark with recently curated questions requiring complex reasoning. Being intuitively reasonable and practically simple, extensive experiments show that agents trained with ReSeek significantly outperform SOTA baselines in task success rate and path faithfulness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑