arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.10552cs.CL
收藏

URaG:多模态大语言模型中统一检索与生成以实现高效长文档理解

URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding

Yongxin Shi, Jiapeng Wang, Zeyu Shan, Dezhi Peng, Zening Lin, Lianwen Jin

AI总结:

针对多模态大语言模型长文档理解的信息干扰与计算成本问题,提出URaG框架,利用其粗到细推理模式统一检索与生成,在单个模型中实现高效证据选择,提升性能并降低44-56%计算开销。

摘要:

近期的多模态大语言模型(MLLMs)在长文档理解方面仍面临挑战,主要源于两大基础问题:大量无关内容的信息干扰,以及基于Transformer架构的二次计算成本。现有方法主要分为两类:牺牲细粒度细节的token压缩;引入外部检索器,增加系统复杂度并阻碍端到端优化。为解决这些问题,我们进行深入分析,观察到MLLMs表现出类人的粗到细推理模式:早期Transformer层广泛关注文档内容,而深层则聚焦于相关证据页。受此启发,我们提出URaG框架,在单个MLLM中统一检索与生成。URaG引入轻量级跨模态检索模块,将早期Transformer层转化为高效证据选择器,识别并保留最相关页面,丢弃无关内容。该设计使深层能集中计算资源于相关信息,同时提升准确性与效率。大量实验表明,URaG实现了最先进性能,且计算开销降低44-56%。代码可在https://github.com/shi-yx/URaG获取。

英文摘要:

Recent multimodal large language models (MLLMs) still struggle with long document understanding due to two fundamental challenges: information interference from abundant irrelevant content, and the quadratic computational cost of Transformer-based architectures. Existing approaches primarily fall into two categories: token compression, which sacrifices fine-grained details; and introducing external retrievers, which increase system complexity and prevent end-to-end optimization. To address these issues, we conduct an in-depth analysis and observe that MLLMs exhibit a human-like coarse-to-fine reasoning pattern: early Transformer layers attend broadly across the document, while deeper layers focus on relevant evidence pages. Motivated by this insight, we posit that the inherent evidence localization capabilities of MLLMs can be explicitly leveraged to perform retrieval during the reasoning process, facilitating efficient long document understanding. To this end, we propose URaG, a simple-yet-effective framework that Unifies Retrieval and Generation within a single MLLM. URaG introduces a lightweight cross-modal retrieval module that converts the early Transformer layers into an efficient evidence selector, identifying and preserving the most relevant pages while discarding irrelevant content. This design enables the deeper layers to concentrate computational resources on pertinent information, improving both accuracy and efficiency. Extensive experiments demonstrate that URaG achieves state-of-the-art performance while reducing computational overhead by 44-56%. The code is available at https://github.com/shi-yx/URaG.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑