arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.11864cs.IRcs.AIcs.CVcs.MM

非常高效的长文档多模态重排序方法

Very Efficient Listwise Multimodal Reranking for Long Documents

Magellan Technology Research Institute (MTRI)(马杰拉技术研究院(MTRI))

Yiqun Sun, Pengfei Wei, Lawrence B. Hsieh

AI总结:

本文提出ZipRerank,通过轻量级查询-图像早期交互机制和单次前向传递消除自回归解码,实现高效多模态重排序,实验表明其在MMDocIR基准上性能优异且显著降低LLM推理延迟。

摘要:

Listwise reranking是视觉中心检索和长文档多模态检索增强生成(M-RAG)中的关键但计算成本高的组件。尽管最近基于VLM的重排序器在准确性上表现强劲,但其实用性常受限于长视觉令牌序列和多步骤自回归解码。我们提出ZipRerank,一种高效的列表式多模态重排序器,直接解决这两个瓶颈。它通过轻量级查询-图像早期交互机制减少输入长度,并通过单次前向传递消除自回归解码。为实现有效学习,ZipRerank采用两阶段训练策略:(i)在大规模文本数据渲染为图像上进行列表式预训练,(ii)使用VLM教师蒸馏软排序监督进行多模态微调。在MMDocIR基准上的广泛实验表明,ZipRerank在性能上与现有最先进多模态重排序器相当或更优,同时将LLM推理延迟降低了一个数量级,使其适用于对延迟敏感的现实系统。代码可在https://github.com/dukesun99/ZipRerank获取。

英文摘要:

Listwise reranking is a key yet computationally expensive component in vision-centric retrieval and multimodal retrieval-augmented generation (M-RAG) over long documents. While recent VLM-based rerankers achieve strong accuracy, their practicality is often limited by long visual-token sequences and multi-step autoregressive decoding. We propose ZipRerank, a highly efficient listwise multimodal reranker that directly addresses both bottlenecks. It reduces input length via a lightweight query-image early interaction mechanism and eliminates autoregressive decoding by scoring all candidates in a single forward pass. To enable effective learning, ZipRerank adopts a two-stage training strategy: (i) listwise pretraining on large-scale text data rendered as images, and (ii) multimodal finetuning with VLM-teacher-distilled soft-ranking supervision. Extensive experiments on the MMDocIR benchmark show that ZipRerank matches or surpasses state-of-the-art multimodal rerankers while reducing LLM inference latency by up to an order of magnitude, making it well-suited for latency-sensitive real-world systems. The code is available at https://github.com/dukesun99/ZipRerank.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑