arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.22240cs.CV
收藏

MMSearch-OPD:将易制难,将难可学习——面向长程多模态搜索智能体

MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

Peking University(北京大学);HKUST(GZ)(香港科技大学(广州));Tsinghua University(清华大学);The University of Hong Kong(香港大学);Zhejiang University(浙江大学);Huawei Cloud BU(华为云业务部)

Huanyao Zhang, Jiepeng Zhou, Jiaye Lin, Ontai Tang, Jingtian Hao, Mugeng Liu, Ru Peng, Lang Mei, Xiaohan Yu, Liyan Liu, Chong Chen, Wentao Zhang

AI总结:

针对长程多模态搜索智能体训练的挑战性问题,提出结合可控数据优化、联合RL与OPD的MMSearch-OPD,经实验验证可提升长程多模态深度搜索能力。

摘要:

多模态深度搜索正从短查询转向持续获取证据的长程调查任务,但这类智能体的训练仍受限于挑战性训练问题不足、过程反馈欠缺。我们提出MMSearch-OPD,将可控数据优化与联合强化学习(RL)及在线策略蒸馏(OPD)相结合。其Refine流程针对现有简单多模态问题衍生的推理链进行定向扩展,参数化变换控制搜索深度与跨模态证据需求,同时复用现有图像-问题对。训练阶段,答案奖励与轻量行为惩罚引导RL,奖励差距调节对不完美学生轨迹的token级教师指导,指数衰减的蒸馏系数随训练推进降低对教师的依赖。实验显示数据优化带来性能提升,经SFT后联合训练进一步改善,表明该方法可提升长程多模态深度搜索能力。

英文摘要:

Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents remains constrained by limited challenging training questions and insufficient process feedback. We introduce MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD). Our Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions. Parameterized transformations control search depth and cross-modal evidence requirements while reusing existing image--question pairs. For training, answer rewards and lightweight behavior penalties guide RL, while the reward gap modulates token-level teacher guidance on imperfect student trajectories. An exponentially decaying distillation coefficient reduces teacher dependence as training proceeds. Experiments show gains from data refinement and further improvements from joint training after SFT, indicating stronger long-horizon multimodal deep search capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑