VAT-KG：面向检索增强生成的知识密集型多模态知识图谱数据集
VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation
AI总结：
针对现有MMKG范围受限问题，提出首个概念中心、知识密集型的VAT-KG，覆盖视觉、音频和文本，并通过严格对齐流程自动构建，结合新RAG框架，实验证明其有效支持MLLM多模态问答。
摘要：
多模态知识图谱（MMKGs）以多种模态表示显式知识，通过补充多模态大语言模型（MLLMs）的隐式知识，并借助检索增强生成（RAG）实现更可靠的推理，发挥着关键作用。然而，现有的多模态知识图谱在范围上普遍受限：它们通常通过扩展现有知识图谱构建，这限制了其知识来源，导致知识覆盖过时或不完整，且往往仅支持有限范围的模态，如文本和视觉信息。这些局限限制了对多模态任务的适用性，尤其是当最新的多模态大语言模型采用视频和音频等更丰富的模态时。因此，我们提出了视觉-音频-文本知识图谱（VAT-KG），这是首个以概念为中心且知识密集型的多模态知识图谱，涵盖视觉、音频和文本信息，其中每个三元组都与多模态数据关联，并附有概念的详细描述。具体而言，我们的构建流程通过一系列严格的过滤和对齐步骤，确保多模态数据与细粒度语义之间的跨模态知识对齐，从而能够从任何多模态数据集自动生成多模态知识图谱。我们进一步引入了一种新颖的多模态RAG框架，该框架针对任意模态的查询检索详细的概念级知识。跨多种模态的问答任务实验证明了VAT-KG在支持多模态大语言模型方面的有效性，凸显了其在统一和利用多模态知识方面的实用价值。
英文摘要：
Multimodal Knowledge Graphs (MMKGs), which represent explicit knowledge across multiple modalities, play a pivotal role by complementing the implicit knowledge of Multimodal Large Language Models (MLLMs) and enabling more grounded reasoning via Retrieval Augmented Generation (RAG). However, existing MMKGs are generally limited in scope: they are often constructed by augmenting pre-existing knowledge graphs, which restricts their knowledge, resulting in outdated or incomplete knowledge coverage, and they often support only a narrow range of modalities, such as text and visual information. These limitations restrict applicability to multimodal tasks, particularly as recent MLLMs adopt richer modalities like video and audio. Therefore, we propose the Visual-Audio-Text Knowledge Graph (VAT-KG), the first concept-centric and knowledge-intensive multimodal knowledge graph that covers visual, audio, and text information, where each triplet is linked to multimodal data and enriched with detailed descriptions of concepts. Specifically, our construction pipeline ensures cross-modal knowledge alignment between multimodal data and fine-grained semantics through a series of stringent filtering and alignment steps, enabling the automatic generation of MMKGs from any multimodal dataset. We further introduce a novel multimodal RAG framework that retrieves detailed concept-level knowledge in response to queries from arbitrary modalities. Experiments on question answering tasks across various modalities demonstrate the effectiveness of VAT-KG in supporting MLLMs, highlighting its practical value in unifying and leveraging multimodal knowledge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
