M^2ConceptBase：一个细粒度对齐的以概念为中心的多模态知识库
M^2ConceptBase: A Fine-Grained Aligned Concept-Centric Multimodal Knowledge Base
AI总结：
本文提出首个以概念为中心的多模态知识库M^2ConceptBase，通过上下文感知符号接地实现概念-图像与描述对齐，提升多模态模型细粒度概念理解及VQA性能。
摘要：
多模态知识库（MMKBs）为多模态任务提供跨模态对齐的知识。然而，现有MMKBs中的图像通常是为百科知识图谱中的实体收集的。因此，缺乏视觉语义与语言概念的详细对应关系，而这对于多模态模型的视觉概念认知能力至关重要。针对这一空白，我们引入了M^2ConceptBase，这是第一个以概念为中心的多模态知识库。M^2ConceptBase将概念建模为节点，并关联图像和详细的文本描述。我们提出了一种上下文感知的多模态符号接地方法，利用图像-文本数据集中的上下文信息来对齐概念-图像和概念-描述对。M^2ConceptBase包含95.1万张图像和15.2万个概念，每个概念平均关联6.27张图像和一个描述，确保全面的视觉和文本语义。人工研究证实对齐准确率超过95%，彰显其质量。此外，我们的实验表明，M^2ConceptBase显著提升了OK-VQA任务上VQA模型的性能。通过两个概念相关任务中的检索增强，M^2ConceptBase还大幅提升了多模态大语言模型的细粒度概念理解能力，凸显其价值。
英文摘要：
Multimodal knowledge bases (MMKBs) provide cross-modal aligned knowledge crucial for multimodal tasks. However, the images in existing MMKBs are generally collected for entities in encyclopedia knowledge graphs. Therefore, detailed groundings of visual semantics with linguistic concepts are lacking, which are essential for the visual concept cognition ability of multimodal models. Addressing this gap, we introduce M^2ConceptBase, the first concept-centric MMKB. M^2ConceptBase models concepts as nodes with associated images and detailed textual descriptions. We propose a context-aware multimodal symbol grounding approach to align concept-image and concept-description pairs using context information from image-text datasets. Comprising 951K images and 152K concepts, M^2ConceptBase links each concept to an average of 6.27 images and a single description, ensuring comprehensive visual and textual semantics. Human studies confirm more than 95% alignment accuracy, underscoring its quality. Additionally, our experiments demonstrate that M^2ConceptBase significantly enhances VQA model performance on the OK-VQA task. M^2ConceptBase also substantially improves the fine-grained concept understanding capabilities of multimodal large language models through retrieval augmentation in two concept-related tasks, highlighting its value.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
