MoLoRAG：通过多模态逻辑感知检索引导文档理解
MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrieval
AI总结：
提出MoLoRAG，一种逻辑感知的多模态检索框架，通过页面图遍历结合语义与逻辑相关性，提升多页文档问答的准确率和检索精度。
摘要：
文档理解是一项基础的人工智能能力，具有广泛的应用，而文档问答（DocQA）是一个关键的评估任务。传统方法将文档转换为文本以供大型语言模型（LLMs）处理，但这一过程会剥离掉如图表等关键的多模态信息。虽然大型视觉语言模型（LVLMs）解决了这一局限性，但其受限的输入大小使得多页文档理解变得不可行。检索增强生成（RAG）方法通过选择相关页面来缓解这一问题，但它们仅依赖语义相关性，忽略了页面与查询之间的逻辑联系，而这对于推理至关重要。为此，我们提出了MoLoRAG，一个用于多模态、多页文档理解的逻辑感知检索框架。通过构建捕获页面间上下文关系的页面图，一个轻量级VLM执行图遍历以检索相关页面，包括那些常被忽略但具有逻辑联系的页面。该方法结合了语义和逻辑相关性，以提供更准确的检索。检索后，将前K个页面输入任意LVLMs进行问答。为了增强灵活性，MoLoRAG提供了两种变体：一种免训练方案，便于部署；一种微调版本，以改进逻辑相关性检查。在四个DocQA数据集上的实验表明，与LVLM直接推理相比，平均准确率提高了9.68%，与基线相比，检索精度提高了7.44%。代码和数据集已在https://github.com/WxxShirley/MoLoRAG发布。
英文摘要：
Document Understanding is a foundational AI capability with broad applications, and Document Question Answering (DocQA) is a key evaluation task. Traditional methods convert the document into text for processing by Large Language Models (LLMs), but this process strips away critical multi-modal information like figures. While Large Vision-Language Models (LVLMs) address this limitation, their constrained input size makes multi-page document comprehension infeasible. Retrieval-augmented generation (RAG) methods mitigate this by selecting relevant pages, but they rely solely on semantic relevance, ignoring logical connections between pages and the query, which is essential for reasoning. To this end, we propose MoLoRAG, a logic-aware retrieval framework for multi-modal, multi-page document understanding. By constructing a page graph that captures contextual relationships between pages, a lightweight VLM performs graph traversal to retrieve relevant pages, including those with logical connections often overlooked. This approach combines semantic and logical relevance to deliver more accurate retrieval. After retrieval, the top-$K$ pages are fed into arbitrary LVLMs for question answering. To enhance flexibility, MoLoRAG offers two variants: a training-free solution for easy deployment and a fine-tuned version to improve logical relevance checking. Experiments on four DocQA datasets demonstrate average improvements of 9.68% in accuracy over LVLM direct inference and 7.44% in retrieval precision over baselines. Codes and datasets are released at https://github.com/WxxShirley/MoLoRAG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
