arXiv 2502.180172025-06-03cs.CVcs.AIcs.CLcs.IR

ViDoRAG：基于动态迭代推理智能体的视觉文档检索增强生成

ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents

MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, USTC（脑启发智能感知与认知联合实验室，中国科学技术大学）；Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团）；Shanghai Jiao Tong University（上海交通大学）

Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, Feng Zhao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视觉丰富文档的复杂推理检索难题，提出多智能体框架ViDoRAG，结合GMM混合检索与探索-总结-反思迭代工作流，在ViDoSeek基准上性能提升超10%。

摘要：

理解视觉丰富文档中的信息仍然是传统检索增强生成（RAG）方法面临的一项重大挑战。现有基准主要聚焦于基于图像的问答（QA），忽视了在密集视觉文档中进行高效检索、理解和推理的根本性挑战。为弥合这一差距，我们引入了ViDoSeek，这是一个新颖的数据集，旨在评估需要复杂推理的视觉丰富文档上的RAG性能。基于该数据集，我们识别出当前RAG方法的关键局限性：（i）纯视觉检索方法难以有效整合文本和视觉特征，以及（ii）先前的方法通常分配不足的推理token，限制了其有效性。为应对这些挑战，我们提出了ViDoRAG，这是一种专为跨视觉文档的复杂推理而设计的新型多智能体RAG框架。ViDoRAG采用基于高斯混合模型（GMM）的混合策略来有效处理多模态检索。为了进一步激发模型的推理能力，我们引入了一个包含探索、总结和反思的迭代智能体工作流程，为研究RAG领域中的测试时扩展提供了框架。在ViDoSeek上的大量实验验证了我们方法的有效性和泛化能力。值得注意的是，ViDoRAG在竞争激烈的ViDoSeek基准上以超过10%的优势超越了现有方法。代码可在https://github.com/Alibaba-NLP/ViDoRAG获取。

英文摘要：

Understanding information from visually rich documents remains a significant challenge for traditional Retrieval-Augmented Generation (RAG) methods. Existing benchmarks predominantly focus on image-based question answering (QA), overlooking the fundamental challenges of efficient retrieval, comprehension, and reasoning within dense visual documents. To bridge this gap, we introduce ViDoSeek, a novel dataset designed to evaluate RAG performance on visually rich documents requiring complex reasoning. Based on it, we identify key limitations in current RAG approaches: (i) purely visual retrieval methods struggle to effectively integrate both textual and visual features, and (ii) previous approaches often allocate insufficient reasoning tokens, limiting their effectiveness. To address these challenges, we propose ViDoRAG, a novel multi-agent RAG framework tailored for complex reasoning across visual documents. ViDoRAG employs a Gaussian Mixture Model (GMM)-based hybrid strategy to effectively handle multi-modal retrieval. To further elicit the model's reasoning capabilities, we introduce an iterative agent workflow incorporating exploration, summarization, and reflection, providing a framework for investigating test-time scaling in RAG domains. Extensive experiments on ViDoSeek validate the effectiveness and generalization of our approach. Notably, ViDoRAG outperforms existing methods by over 10% on the competitive ViDoSeek benchmark. The code is available at https://github.com/Alibaba-NLP/ViDoRAG.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。