KG-ViP：在多模态大语言模型中桥接知识基础与视觉感知以进行视觉问答
KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering
AI总结：
提出KG-ViP框架，通过检索与融合场景图和常识图，统一外部知识与细粒度视觉细节，缓解多模态大语言模型在视觉问答中的知识幻觉和视觉感知不足问题。
摘要：
用于视觉问答（VQA）的多模态大语言模型（MLLMs）通常面临双重限制：知识幻觉和细粒度视觉感知不足。关键的是，我们发现常识图和场景图通过提供丰富的外部知识和捕捉细粒度视觉细节，恰好为这些缺陷提供了互补的解决方案。然而，先前的工作通常孤立地处理它们，忽视了它们的协同潜力。为了弥合这一差距，我们提出了KG-ViP，一个统一的框架，通过融合场景图和常识图来增强MLLMs。KG-ViP框架的核心是一个新颖的检索与融合流程，利用查询作为语义桥逐步整合两种图，合成统一的结构化上下文，促进可靠的多模态推理。在FVQA 2.0+和MVQA基准上的大量实验表明，KG-ViP显著优于现有的VQA方法。
英文摘要：
Multi-modal Large Language Models (MLLMs) for Visual Question Answering (VQA) often suffer from dual limitations: knowledge hallucination and insufficient fine-grained visual perception. Crucially, we identify that commonsense graphs and scene graphs provide precisely complementary solutions to these respective deficiencies by providing rich external knowledge and capturing fine-grained visual details. However, prior works typically treat them in isolation, overlooking their synergistic potential. To bridge this gap, we propose KG-ViP, a unified framework that empowers MLLMs by fusing scene graphs and commonsense graphs. The core of the KG-ViP framework is a novel retrieval-and-fusion pipeline that utilizes the query as a semantic bridge to progressively integrate both graphs, synthesizing a unified structured context that facilitates reliable multi-modal reasoning. Extensive experiments on FVQA 2.0+ and MVQA benchmarks demonstrate that KG-ViP significantly outperforms existing VQA methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
