arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.15470cs.IRcs.MA
收藏

CogPlanner:通过规划揭示智能体多模态检索增强生成的潜力

CogPlanner: Unveiling the Potential of Agentic Multimodal Retrieval Augmented Generation with Planning

Huawei Cloud BU(华为云业务单元)

Xiaohan Yu, Zhihan Yang, Chong Chen

AI总结:

本文提出MRAG Planning任务及受人类认知启发的即插即用框架CogPlanner,通过迭代确定查询重构与检索策略,并引入CogBench基准,在最小计算开销下显著提升了多模态检索增强生成的准确性与效率。

摘要:

多模态检索增强生成(MRAG)系统在增强多模态大语言模型(MLLMs)的生成能力方面展现出了潜力。然而,现有的MRAG框架主要遵循僵化的单步检索策略,无法应对现实世界中信息获取和查询重构的挑战。在这项工作中,我们引入了多模态检索增强生成规划(MRAG Planning)任务,旨在有效获取和整合信息的同时最小化计算开销。具体而言,我们提出了CogPlanner,一个受人类认知过程启发的智能体即插即用框架,它迭代地确定查询重构和检索策略,以生成准确且与上下文相关的响应。CogPlanner支持并行和顺序建模范式。此外,我们引入了CogBench,这是一个旨在严格评估MRAG Planning任务并促进将轻量级CogPlanner与资源高效的MLLMs(如Qwen2-VL-7B-Cog)集成的新基准。实验结果表明,CogPlanner显著优于现有的MRAG基线,在最小的额外计算成本下,在准确性和效率两方面均提供了改进。

英文摘要:

Multimodal Retrieval Augmented Generation (MRAG) systems have shown promise in enhancing the generation capabilities of multimodal large language models (MLLMs). However, existing MRAG frameworks primarily adhere to rigid, single-step retrieval strategies that fail to address real-world challenges of information acquisition and query reformulation. In this work, we introduce the task of Multimodal Retrieval Augmented Generation Planning (MRAG Planning) that aims at effective information seeking and integration while minimizing computational overhead. Specifically, we propose CogPlanner, an agentic plug-and-play framework inspired by human cognitive processes, which iteratively determines query reformulation and retrieval strategies to generate accurate and contextually relevant responses. CogPlanner supports parallel and sequential modeling paradigms. Furthermore, we introduce CogBench, a new benchmark designed to rigorously evaluate the MRAG Planning task and facilitate lightweight CogPlanner integration with resource-efficient MLLMs, such as Qwen2-VL-7B-Cog. Experimental results demonstrate that CogPlanner significantly outperforms existing MRAG baselines, offering improvements in both accuracy and efficiency with minimal additional computational costs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑