MARVEL:多模态自适应推理-增强扩展-排序和检索
MARVEL: Multimodal Adaptive Reasoning-intensiVe Expand-rerank and retrievaL
AI总结:
本文提出MARVEL框架,通过统一的扩展-检索-排序流程,结合LLM驱动的查询扩展、增强推理的密集检索器和基于GPT-4o的逐步推理排序,提升了多模态检索性能,在MM-BRIGHT基准上实现了37.9nDCG@10的高分。
摘要:
多模态文本库检索仍是一个基本挑战:最好的视觉-语言编码器在MM-BRIGHT基准上仅能获得27.6nDCG@10,低于强文本-only系统。我们主张有效的多模态检索需要三个紧密集成的能力,现有方法仅单独处理这些能力:扩展查询的潜在意图、使用为复杂推理训练的模型进行检索、以及通过显式的逐步推理对候选进行排序。我们引入MARVEL(多模态自适应推理-增强扩展-排序和检索),一个统一的流程,结合LLM驱动的查询扩展、MARVEL-Retriever——一个为复杂多模态查询微调的增强推理密集检索器,以及基于GPT-4o的逐步推理排序,带有可选的多轮递归排名融合。在MM-BRIGHT上评估29个技术领域,MARVEL实现了37.9nDCG@10,超越最佳多模态编码器10.3分,并在27个领域中优于所有单阶段基线,在剩余两个高度专业化的领域(加密、量子计算)中达到或接近最佳基线,证明了通过统一的扩展-检索-排序框架处理推理密集的多模态检索是最佳方法。
英文摘要:
Multimodal retrieval over text corpora remains a fundamental challenge: the best vision-language encoder achieves only 27.6 nDCG@10 on MM-BRIGHT, a reasoning-intensive multimodal retrieval benchmark, underperforming strong text-only systems. We argue that effective multimodal retrieval requires three tightly integrated capabilities that existing approaches address only in isolation: expanding the query's latent intent, retrieving with a model trained for complex reasoning, and reranking via explicit step-by-step reasoning over candidates. We introduce \textbf{MARVEL} (\textbf{M}ultimodal \textbf{A}daptive \textbf{R}easoning-intensi\textbf{V}e \textbf{E}xpand-rerank and retrieva\textbf{L}), a unified pipeline that combines LLM-driven query expansion, \textbf{MARVEL-Retriever} -- a reasoning-enhanced dense retriever fine-tuned for complex multimodal queries -- and GPT-4o-based chain-of-thought reranking with optional multi-pass reciprocal rank fusion. Evaluated on MM-BRIGHT across 29 technical domains, MARVEL achieves \textbf{37.9} nDCG@10, surpassing the best multimodal encoder by \textbf{+10.3 points} and outperforming all single-stage baselines in 27 of 29 domains and matching or approaching the best baseline in the remaining two highly-specialized domains (Crypto, Quantum Computing), demonstrating that reasoning-intensive multimodal retrieval is best addressed through a unified expand-retrieve-rerank framework. https://github.com/mm-bright/multimodal-reasoning-retrieval
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。