MM-BRIGHT: 一种多任务多模态基准用于推理密集型检索
MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval
AI总结:
MM-BRIGHT是首个用于推理密集型检索的多模态基准,通过29个技术领域中的2,803个现实世界查询,评估了多模态检索模型在文本到文本、多模态到文本等任务上的性能。
摘要:
现有的检索基准主要由基于文本的查询组成,其中关键词或语义匹配通常就足够了。许多现实世界的查询包含多模态元素,特别是图像,如图表和截图,这些需要密集的推理才能识别相关文档。为了解决这一差距,我们引入了MM-BRIGHT,这是第一个用于推理密集型检索的多模态基准。我们的数据集包含2,803个现实世界的查询,涵盖29个不同的技术领域,有四个逐渐增加复杂性的任务:文本到文本、多模态到文本、多模态到图像以及多模态到多模态检索。广泛的评估显示,最先进的模型在所有任务上都面临挑战:BM25在纯文本检索中仅达到8.5 nDCG@10,而最好的多模态模型Nomic-Vision在多模态到文本检索中仅达到27.6 nDCG@10,实际上甚至不如最好的纯文本模型(DiVeR: 32.2)。这些结果突显了巨大的提升空间,并将MM-BRIGHT定位为下一代检索模型的测试平台,这些模型能更好地整合视觉推理。我们的代码和数据可在https://github.com/mm-bright/MM-BRIGHT上获得。另见我们的官方网站:https://mm-bright.github.io/。
英文摘要:
Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce MM-BRIGHT, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval. Extensive evaluation reveals that state-of-the-art models struggle across all tasks: BM25 achieves only 8.5 nDCG@10 on text-only retrieval, while the best multimodal model Nomic-Vision reaches just 27.6 nDCG@10 on multimodal-to-text retrieval actually underperforming the best text-only model (DiVeR: 32.2). These results highlight substantial headroom and position MM-BRIGHT as a testbed for next-generation retrieval models that better integrate visual reasoning. Our code and data are available at https://github.com/mm-bright/MM-BRIGHT. See also our official website: https://mm-bright.github.io/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。