索引轻,推理深:延迟视觉摄入用于视觉密集文档问答
Index Light, Reason Deep: Deferred Visual Ingestion for Visual-Dense Document Question Answering
AI总结:
本文提出延迟视觉摄入框架,通过分层图号聚类构建索引,利用BM25检索和VLM分析提升视觉密集文档问答性能。
摘要:
现有的多模态文档问答方法大多采用预摄入(PI)策略:在索引阶段,对每一页调用视觉语言模型(VLM)生成页面描述,然后将这些描述编码成向量,问题通过嵌入相似性检索来回答。然而,这种方法在视觉密集的工程文档中面临双重困境:VLM盲目的描述不可避免地会丢失关键的视觉细节,而嵌入检索在高度相似的文档上系统性地失败。本文提出了延迟视觉摄入(DVI)框架:预处理阶段不调用任何VLM,仅利用文档结构信息(目录、图号)通过HDNC(分层图号聚类)算法自动构建分层索引;在推理阶段,通过BM25检索定位候选页面,然后将原始图像以及具体问题发送给VLM进行针对性分析。在三个数据集上的大规模实验验证了DVI的有效性:在桥梁工程图纸(1,323个问题)上,端到端QA准确率达到65.6% vs PI的24.3%(+41.3个百分点);在钢目录(186个问题)上,30.6% vs 16.1%(+14.5个百分点);在CircuitVQA公共基准(9,315个问题)上,检索ImgR@3达到31.2% vs 0.7%。在桥梁数据集上,我们评估了ColPali(ICLR 2025视觉检索SOTA),其仅达到20.1% PageR@3,证明在同质工程文档上嵌入检索的失败是结构问题而非模型能力不足。消融研究显示,HDNC零成本自动索引带来+27.5个百分点的检索提升,VLM转换率分析确认瓶颈在于检索而非理解方面。
英文摘要:
Existing multimodal document question answering methods predominantly adopt a Pre-Ingestion (PI) strategy: during the indexing phase, a Vision Language Model (VLM) is called on every page to generate page descriptions that are then encoded into vectors, and questions are answered via embedding similarity retrieval. However, this approach faces a dual dilemma on visual-dense engineering documents: VLM blind descriptions inevitably lose critical visual details, and embedding retrieval systematically fails on highly similar documents. This paper proposes the Deferred Visual Ingestion (DVI) framework: zero VLM calls during preprocessing, leveraging only document structural information (table of contents, drawing numbers) to automatically build a hierarchical index through the HDNC (Hierarchical Drawing Number Clustering) algorithm; during inference, candidate pages are located via BM25 retrieval, and the original images along with the specific question are sent to a VLM for targeted analysis. Large-scale experiments on three datasets validate the effectiveness of DVI: on Bridge engineering drawings (1,323 questions), end-to-end QA accuracy reaches 65.6\% vs. PI's 24.3\% (+41.3pp); on Steel catalog (186 questions), 30.6\% vs. 16.1\% (+14.5pp); on CircuitVQA, a public benchmark (9,315 questions), retrieval ImgR@3 achieves 31.2\% vs. 0.7\%. On the Bridge dataset, we evaluated ColPali (ICLR 2025 visual retrieval SOTA), which achieved only 20.1\% PageR@3, demonstrating that the failure of embedding retrieval on homogeneous engineering documents is structural rather than due to insufficient model capability. Ablation studies show that HDNC zero-cost automatic indexing yields a +27.5pp retrieval improvement, and VLM conversion rate analysis confirms that the bottleneck lies on the retrieval side rather than the comprehension side.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。