M$^3$-VQA：多模态、多实体、多跳视觉问答的基准测试
M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
AI总结：
M$^3$-VQA引入了多实体问题，要求模型在多个文档间进行顺序和并行多跳推理，揭示了多模态大语言模型在知识获取和推理方面的挑战。
摘要：
我们提出了M$^3$-VQA，一个基于知识的视觉问答（VQA）基准测试，旨在提升多模态大语言模型（MLLMs）在细粒度多模态实体理解和复杂多跳推理中的评估。与现有VQA数据集专注于粗粒度类别和单实体简单推理不同，M$^3$-VQA引入了涉及多个不同实体的多样化多实体问题，要求模型在多个文档间进行顺序和并行多跳推理，支持可追溯的详细证据和一个精心编纂的多模态知识库。我们评估了16种领先的MLLMs在三种设置下的表现：无外部知识、有黄金证据和检索增强输入。糟糕的结果揭示了MLLMs在知识获取和推理方面的重大挑战。模型在无外部信息时表现不佳，但提供精确证据时显著改善。此外，推理意识的代理检索超越了启发式方法，突显了结构化推理对于复杂多模态理解的重要性。M$^3$-VQA为提升MLLMs的多模态推理能力提供了更严峻的评估。我们的代码和数据集可在https://github.com/CASIA-IVA-Lab/M3VQA获取。
英文摘要：
We present M$^3$-VQA, a novel knowledge-based Visual Question Answering (VQA) benchmark, to enhance the evaluation of multimodal large language models (MLLMs) in fine-grained multimodal entity understanding and complex multi-hop reasoning. Unlike existing VQA datasets that focus on coarse-grained categories and simple reasoning over single entities, M$^3$-VQA introduces diverse multi-entity questions involving multiple distinct entities from both visual and textual sources. It requires models to perform both sequential and parallel multi-hop reasoning across multiple documents, supported by traceable, detailed evidence and a curated multimodal knowledge base. We evaluate 16 leading MLLMs under three settings: without external knowledge, with gold evidence, and with retrieval-augmented input. The poor results reveal significant challenges for MLLMs in knowledge acquisition and reasoning. Models perform poorly without external information but improve markedly when provided with precise evidence. Furthermore, reasoning-aware agentic retrieval surpasses heuristic methods, highlighting the importance of structured reasoning for complex multimodal understanding. M$^3$-VQA presents a more challenging evaluation for advancing the multimodal reasoning capabilities of MLLMs. Our code and dataset are available at https://github.com/CASIA-IVA-Lab/M3VQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
