arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02185cs.CVcs.AIcs.CLcs.LG
收藏

视觉-深度研究基准:重新思考多模态大语言模型的视觉和文本搜索

Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models

Yu Zeng, Wenxuan Huang, Zhen Fang, Shuang Chen, Yufan Shen, Yishuo Cai, Xiaoman Wang, Zhenfei Yin, Lin Chen, Zehui Chen, Shiting Huang, Yiming Zhao, Xu Tang, Ya… 展开作者

Yu Zeng, Wenxuan Huang, Zhen Fang, Shuang Chen, Yufan Shen, Yishuo Cai, Xiaoman Wang, Zhenfei Yin, Lin Chen, Zehui Chen, Shiting Huang, Yiming Zhao, Xu Tang, Yao Hu, Philip Torr, Wanli Ouyang, Shaosheng Cao

AI总结:

本文提出Vision-DeepResearch基准,通过精心设计的问题评估多模态大语言模型的视觉与文本搜索能力,并引入多轮裁剪搜索流程提升实际检索性能。

摘要:

多模态大语言模型(MLLMs)已推进了视觉问答(VQA)并现在支持使用搜索引擎进行复杂视觉-文本事实查找的Vision-DeepResearch系统。然而,评估这些视觉和文本搜索能力仍然困难,现有基准有两个主要限制。首先,现有基准不是视觉搜索导向的:需要视觉搜索的答案通常通过文本问题中的跨文本线索泄露,或可以通过当前MLLMs中的先验世界知识推断。其次,过于理想化的评估场景:在图像搜索方面,所需信息通常可以通过与完整图像的近似匹配获得,而文本搜索方面过于直接且不够具有挑战性。为了解决这些问题,我们构建了包含2000个VQA实例的Vision-DeepResearch基准(VDR-Bench)。所有问题均通过精心设计的多阶段筛选流程和严格专家审查创建,旨在在现实世界条件下评估Vision-DeepResearch系统的行为。此外,为了解决当前MLLMs的视觉检索能力不足问题,我们提出了一种简单的多轮裁剪搜索流程。该策略在现实视觉检索场景中被证明能有效提高模型性能。总体而言,我们的结果为未来多模态深度研究系统的设计提供了实用指导。代码将在https://github.com/Osilly/Vision-DeepResearch上发布。

英文摘要:

Multimodal Large Language Models (MLLMs) have advanced VQA and now support Vision-DeepResearch systems that use search engines for complex visual-textual fact-finding. However, evaluating these visual and textual search abilities is still difficult, and existing benchmarks have two major limitations. First, existing benchmarks are not visual search-centric: answers that should require visual search are often leaked through cross-textual cues in the text questions or can be inferred from the prior world knowledge in current MLLMs. Second, overly idealized evaluation scenario: On the image-search side, the required information can often be obtained via near-exact matching against the full image, while the text-search side is overly direct and insufficiently challenging. To address these issues, we construct the Vision-DeepResearch benchmark (VDR-Bench) comprising 2,000 VQA instances. All questions are created via a careful, multi-stage curation pipeline and rigorous expert review, designed to assess the behavior of Vision-DeepResearch systems under realistic real-world conditions. Moreover, to address the insufficient visual retrieval capabilities of current MLLMs, we propose a simple multi-round cropped-search workflow. This strategy is shown to effectively improve model performance in realistic visual retrieval scenarios. Overall, our results provide practical guidance for the design of future multimodal deep-research systems. The code will be released in https://github.com/Osilly/Vision-DeepResearch.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑