ProMMSearchAgent:一种通过过程导向奖励训练的通用多模态搜索代理
ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards
AI总结:
本文提出ProMMSearchAgent,通过过程导向奖励解决多模态代理在知识密集型视觉推理中的训练难题,实现零样本迁移至Google搜索API。
摘要:
通过强化学习训练多模态代理进行知识密集型视觉推理受到结果导向监督极稀疏性和实时网络环境不可预测性的限制。为解决这些算法和环境瓶颈,我们引入ProMMSearchAgent,建立了一种新的仿真到现实训练范式用于多模态搜索。我们将策略学习分解为确定性的本地静态沙盒。关键在于,为了在受限制的环境中有效学习,我们提出了一种反思性过程导向奖励。通过探测代理自身的参数知识边界,我们生成密集的行为元数据,明确奖励正确的认知决策,仅在视觉或事实不确定时启动多模态或文本搜索。大量实验表明,我们本地训练的策略能够零样本迁移至实时Google搜索API。ProMMSearchAgent实现了新的SOTA性能,在FVQA-test上比MMSearch-R1高出5.1%,在InfoSeek上高出6.3%,在MMSearch上高出11.3%。
英文摘要:
Training multimodal agents via reinforcement learning for knowledge-intensive visual reasoning is fundamentally hindered by the extreme sparsity of outcome-based supervision and the unpredictability of live web environments. To resolve these algorithmic and environmental bottlenecks, we introduce ProMMSearchAgent, establishing a novel Sim-to-Real training paradigm for multimodal search. We decouple policy learning into a deterministic, local static sandbox. Crucially, to learn effectively within this constrained environment, we propose an introspective process-oriented reward. By probing the agent's own parametric knowledge boundaries, we generate dense behavioral metadata that explicitly rewards the correct cognitive decision, initiating a multimodal or text search only when visually or factually uncertain. Extensive experiments demonstrate that our locally-trained policy transfers zero-shot to the live Google Search API. ProMMSearchAgent achieves new SOTA performance, outperforming MMSearch-R1 by +5.1% on FVQA-test, +6.3% on InfoSeek, and +11.3% on MMSearch.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。