VisDoM：使用多模态检索增强生成处理含视觉丰富元素的多文档问答
VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation
AI总结：
针对含视觉丰富元素的多文档问答问题，本文构建VisDoMBench基准，提出融合视觉与文本检索的多模态RAG方法VisDoMRAG，通过多步推理和模态融合机制，在该基准上优于基线12-20%。
摘要：
理解来自多个文档集合的信息，尤其是那些包含视觉丰富元素的文档，对于基于文档的问答至关重要。本文介绍了VisDoMBench，这是首个综合基准，旨在评估多文档场景下具有丰富多模态内容（包括表格、图表和演示幻灯片）的问答系统。我们提出VisDoMRAG，一种新颖的多模态检索增强生成（RAG）方法，同时利用视觉和文本RAG，将强大的视觉检索能力与复杂的语言推理相结合。VisDoMRAG采用多步推理过程，包括证据整理和思维链推理，用于同时进行文本和视觉推理。VisDoMRAG的一个关键创新是其一致性约束的模态融合机制，该机制在推理时对齐跨模态的推理过程，以生成连贯的最终答案。这提高了关键信息分布在多模态场景中的准确性，并通过隐式上下文归因提高了答案的可验证性。通过涉及开源和专有大型语言模型的广泛实验，我们在VisDoMBench上对最先进的文档问答方法进行了基准测试。广泛的结果表明，VisDoMRAG在端到端多模态文档问答中优于单模态和长上下文LLM基线12-20%。
英文摘要：
Understanding information from a collection of multiple documents, particularly those with visually rich elements, is important for document-grounded question answering. This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings with rich multimodal content, including tables, charts, and presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG, combining robust visual retrieval capabilities with sophisticated linguistic reasoning. VisDoMRAG employs a multi-step reasoning process encompassing evidence curation and chain-of-thought reasoning for concurrent textual and visual RAG pipelines. A key novelty of VisDoMRAG is its consistency-constrained modality fusion mechanism, which aligns the reasoning processes across modalities at inference time to produce a coherent final answer. This leads to enhanced accuracy in scenarios where critical information is distributed across modalities and improved answer verifiability through implicit context attribution. Through extensive experiments involving open-source and proprietary large language models, we benchmark state-of-the-art document QA methods on VisDoMBench. Extensive results show that VisDoMRAG outperforms unimodal and long-context LLM baselines for end-to-end multimodal document QA by 12-20%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
