Visual-RAG：面向视觉知识密集型查询的文本到图像检索增强生成基准测试
Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries
AI总结：
Visual-RAG基准通过要求文本到图像检索并整合视觉证据来评估多模态大语言模型在视觉知识密集型问答中的表现，发现现有模型仍难以高效利用视觉知识。
摘要：
检索增强生成（RAG）是一种利用外部知识增强大语言模型（LLM）以处理知识密集型问答的范式。虽然已有若干基准在多模态RAG设置下评估多模态大语言模型（MLLM），但它们主要从文本语料库中检索，并未明确评估模型在生成过程中如何利用视觉证据。因此，目前仍缺乏能够隔离并衡量检索图像在RAG中贡献的基准。我们提出Visual-RAG，一个针对视觉 grounding 的知识密集型问题的问答基准。与先前工作不同，Visual-RAG要求进行文本到图像检索，并整合检索到的线索图像以提取视觉证据用于答案生成。利用Visual-RAG，我们评估了5个开源和3个专有MLLM，结果表明图像在增强生成中提供了强有力的证据。然而，即使是最先进的模型也难以高效地提取和利用视觉知识。我们的结果凸显了在多模态RAG系统中改进视觉检索、grounding和归因的必要性。
英文摘要：
Retrieval-augmented generation (RAG) is a paradigm that augments large language models (LLMs) with external knowledge to tackle knowledge-intensive question answering. While several benchmarks evaluate Multimodal LLMs (MLLMs) under Multimodal RAG settings, they predominantly retrieve from textual corpora and do not explicitly assess how models exploit visual evidence during generation. Consequently, there still lacks benchmark that isolates and measures the contribution of retrieved images in RAG. We introduce Visual-RAG, a question-answering benchmark that targets visually grounded, knowledge-intensive questions. Unlike prior work, Visual-RAG requires text-to-image retrieval and the integration of retrieved clue images to extract visual evidence for answer generation. With Visual-RAG, we evaluate 5 open-source and 3 proprietary MLLMs, showcasing that images provide strong evidence in augmented generation. However, even state-of-the-art models struggle to efficiently extract and utilize visual knowledge. Our results highlight the need for improved visual retrieval, grounding, and attribution in multimodal RAG systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
