arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.22278cs.IRcs.LG
收藏

RETLLM: 无需训练和数据的多模态信息检索中的大规模语言模型训练

RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval

College of Computer Science;Software Engineering \ University, Shenzhen, China

Dawei Su, Dongsheng Wang

AI总结:

RetLLM通过无需训练和数据的框架,利用MLLMs的多模态推理能力提升多模态信息检索性能。

摘要:

多模态信息检索(MMIR)因其在处理文本、图像或混合查询和候选项方面的灵活性而受到关注。最近在多模态大语言模型(MLLMs)中的突破通过在对比微调框架下整合MLLM知识来提升MMIR性能。然而,它们面临预训练不一致性和需要大量数据集的问题。在本工作中,我们引入了一个新的框架RetLLM,旨在以无需训练和数据的方式查询MLLMs进行MMIR。具体而言,我们将MMIR制定为相似性分数生成任务,并提示MLLMs在粗略-然后-精细的流程中直接预测检索分数。在粗略阶段,一个top-k过滤策略为每个查询构建一个小型但高质量的候选池,使MLLMs能够专注于语义相关的候选。随后,通过将查询和候选输入MLLMs来预测检索分数。重要的是,我们提出了一种在推理过程中增强视觉的模块,帮助MLLMs重新选择被遗忘的视觉信息,从而提高检索效果。在MMIR基准上的广泛实验表明,RetLLM优于微调模型。消融研究进一步验证了每个组件。我们的工作表明,MLLMs可以在没有任何训练的情况下实现强大的MMIR性能,突显了其在简单、可扩展框架中固有的多模态推理能力。我们发布代码地址:https://github.com/alivecat05/RETLLM

英文摘要:

Multimodal information retrieval (MMIR) has gained attention for its flexibility in handling text, images, or mixed queries and candidates. Recent breakthroughs in multimodal large language models (MLLMs) boost MMIR performance by incorporating MLLM knowledge under the contrastive finetuning framework. However, they suffer from pre-training inconsistency and require large datasets. In this work, we introduce a novel framework, RetLLM, designed to query MLLMs for MMIR in a training- and data-free manner. Specifically, we formulate MMIR as a similarity score generation task and prompt MLLMs to directly predict retrieval scores in a coarse-then-fine pipeline. At the coarse stage, a top-k filtering strategy builds a small yet high-quality candidate pool for each query, enabling MLLMs to focus on semantically relevant candidates. Subsequently, the retrieval score is predicted by feeding both the query and candidate into MLLMs at the fine stage. Importantly, we propose a visual enhancement module during reasoning to help MLLMs re-pick forgotten visuals, improving retrieval. Extensive experiments on MMIR benchmarks show that RetLLM outperforms fine-tuned models. Ablation studies further verify each component. Our work demonstrates that MLLMs can achieve strong MMIR performance without any training, highlighting their inherent multimodal reasoning ability in a simple, scalable framework. We release our code at: https://github.com/alivecat05/RETLLM

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑