保障地板，提升天花板：一种基于合并的多模态搜索代理范式
Securing the Floor and Raising the Ceiling: A Merging-based Paradigm for Multi-modal Search Agents
AI总结：
本文提出一种无需训练的多模态搜索代理范式，通过跨模态模型合并提升VLMs的自主搜索能力，OBM算法在搜索任务中表现出更高的效率和性能。
摘要：
近期视觉-语言模型（VLMs）的进步推动了多模态搜索代理的发展，这些代理能够主动调用外部搜索工具并通过多步骤推理整合检索到的证据。尽管有前景，现有方法通常依赖于大规模监督轨迹或昂贵的强化学习（RL），导致训练成本高、不稳定，并且对标准VLMs存在严重的冷启动问题。我们提出了一种无需训练的范式，通过跨模态模型合并来赋予VLMs自主搜索能力。通过融合基于文本的搜索代理与基础VLM，我们证明多模态搜索能力可以在没有额外多模态训练数据的情况下有效组合。为减轻跨模态整合期间的参数干扰，我们引入了最优脑合并（OBM），一种基于显著性的合并算法，通过仅使用少量校准样本，根据参数对模型损失的影响来识别任务关键参数。在搜索密集型基准（如InfoSeek、MMSearch）上的广泛实验表明：（1）模型合并为零样本代理提供了合理的性能地板，OBM实现了更优的搜索速率；（2）OBM作为热启动策略显著提升了性能天花板，比标准VLM初始化更快收敛并达到更高的峰值精度。
英文摘要：
Recent advances in Vision-Language Models (VLMs) have motivated the development of multi-modal search agents that can actively invoke external search tools and integrate retrieved evidence through multi-step reasoning. While promising, existing approaches typically rely on large-scale supervised trajectories or expensive reinforcement learning (RL), leading to high training cost, instability, and a severe cold-start problem for standard VLMs. We propose a training-free paradigm to empower VLMs with autonomous search capabilities via cross-modal model merging. By fusing a text-based search agent with a base VLM, we show that multi-modal search capabilities can be effectively composed without any additional multi-modal training data. To mitigate parameter interference during cross-modal integration, we introduce Optimal Brain Merging (OBM), a saliency-aware merging algorithm that identifies task-critical parameters based on their impact on model loss using only a small set of calibration samples. Extensive experiments on search-intensive benchmarks (e.g., InfoSeek, MMSearch) reveal that: (1) Model merging secures a reasonable performance floor as a zero-shot agent, with OBM achieving superior search rates; (2) OBM significantly raises the performance ceiling as a warm-start strategy, achieving faster convergence and higher peak accuracy than standard VLM initialization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
