arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.15041cs.AIcs.CL
收藏

mR²AG:用于基于知识的视觉问答的多模态检索-反思增强生成

mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information(多模态信息超级智能安全北京市重点实验室);Tencent Inc.(腾讯公司);Huawei Noah’s Ark Laboratory(华为诺亚方舟实验室)

Tao Zhang, Ziqi Zhang, Zongyang Ma, Yuxin Chen, Zhongang Qi, Chunfeng Yuan, Bing Li, Junfu Pu, Yuxuan Zhao, Zehua Xie, Jin Ma, Ying Shan, Weiming Hu

AI总结:

该研究针对多模态大语言模型在基于知识的视觉问答中的缺陷,提出mR²AG框架,通过两种反思操作实现自适应检索与信息定位,在相关基准任务上性能优于现有方法。

摘要:

先进的多模态大语言模型(MLLMs)在近期的基于知识的视觉问答(VQA)任务(如INFOSEEK和Encyclopedic-VQA)中表现不佳,原因在于其知识范围有限且固定,常导致模糊且不准确的回答。因此,多模态检索增强生成(mRAG)被自然引入,为MLLMs提供全面且最新的知识,有效扩展其知识范围。然而,当前的mRAG方法存在固有缺陷:1)即使不需要外部知识也会执行检索;2)无法识别支持查询的证据;3)因额外的信息过滤模块或规则而增加模型复杂度。为解决这些不足,我们提出一种名为多模态检索-反思增强生成(mR²AG)的新型通用框架,该框架通过两种易于实现的反思操作实现自适应检索和有用信息定位,避免高模型复杂度。在mR²AG中,检索-反思旨在区分不同用户查询,避免冗余检索调用;相关性-反思则用于引导MLLM定位检索内容的有益证据并据此生成回答。此外,mR²AG可集成到任何训练良好的MLLM中,仅需在提出的mR²AG指令调优数据集(mR²AG-IT)上进行高效微调。mR²AG在INFOSEEK和Encyclopedic-VQA上显著优于最先进的MLLMs(如GPT-4o)和基于mRAG的MLLMs,同时在广泛的视觉相关任务中保持基础MLLMs的出色能力。

英文摘要:

Advanced Multimodal Large Language Models (MLLMs) struggle with recent Knowledge-based Visual Question Answering (VQA) tasks, such as INFOSEEK and Encyclopedic-VQA, due to their limited and frozen knowledge scope, often leading to ambiguous and inaccurate responses. Thus, multimodal Retrieval-Augmented Generation (mRAG) is naturally introduced to provide MLLMs with comprehensive and up-to-date knowledge, effectively expanding the knowledge scope. However, current mRAG methods have inherent drawbacks, including: 1) Performing retrieval even when external knowledge is not needed. 2) Lacking of identification of evidence that supports the query. 3) Increasing model complexity due to additional information filtering modules or rules. To address these shortcomings, we propose a novel generalized framework called \textbf{m}ultimodal \textbf{R}etrieval-\textbf{R}eflection-\textbf{A}ugmented \textbf{G}eneration (mR$^2$AG), which achieves adaptive retrieval and useful information localization to enable answers through two easy-to-implement reflection operations, preventing high model complexity. In mR$^2$AG, Retrieval-Reflection is designed to distinguish different user queries and avoids redundant retrieval calls, and Relevance-Reflection is introduced to guide the MLLM in locating beneficial evidence of the retrieved content and generating answers accordingly. In addition, mR$^2$AG can be integrated into any well-trained MLLM with efficient fine-tuning on the proposed mR$^2$AG Instruction-Tuning dataset (mR$^2$AG-IT). mR$^2$AG significantly outperforms state-of-the-art MLLMs (e.g., GPT-4o) and mRAG-based MLLMs on INFOSEEK and Encyclopedic-VQA, while maintaining the exceptional capabilities of base MLLMs across a wide range of Visual-dependent tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑