MLLM 是一个强大的重排序器：通过知识增强重排序与噪声注入训练推进多模态检索增强生成
MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training
AI总结：
本文提出 RagVL 框架，通过知识增强重排序机制将 MLLM 作为重排序器过滤检索图像，并在训练中注入视觉噪声以增强生成器鲁棒性，有效解决了多模态检索增强生成中的多粒度噪声对应问题。
摘要：
多模态大语言模型（MLLMs）在处理和生成跨多种数据模态的内容方面展现了卓越的能力。然而，MLLMs 的一个显著缺陷是其依赖静态训练数据，导致信息过时且上下文感知受限。这种静态特性阻碍了其提供准确且最新回复的能力，尤其是在动态或快速演变的上下文中。尽管集成多模态检索增强生成（Multimodal RAG）提供了一个有前景的解决方案，但系统不可避免地会遇到多粒度噪声对应（MNC）问题，这阻碍了准确的检索与生成。在本研究中，我们提出了 RagVL，一个具备知识增强重排序和噪声注入训练的新颖框架，以解决这些局限性。我们使用一个简单而有效的指令模板对 MLLM 进行指令微调，以激发其排序能力，并将其作为重排序器来精确过滤 top-k 检索到的图像。在生成阶段，我们在训练期间在数据层和 token 层注入视觉噪声，以增强生成器的鲁棒性。在两个数据集的子集上进行的广泛实验验证了我们方法的有效性，这些子集要求检索图像并对其进行推理以回答给定查询。代码和模型可在 https://github.com/IDEA-FinAI/RagVL 获取。
英文摘要：
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in processing and generating content across multiple data modalities. However, a significant drawback of MLLMs is their reliance on static training data, leading to outdated information and limited contextual awareness. This static nature hampers their ability to provide accurate and up-to-date responses, particularly in dynamic or rapidly evolving contexts. Though integrating Multimodal Retrieval-augmented Generation (Multimodal RAG) offers a promising solution, the system would inevitably encounter the multi-granularity noisy correspondence (MNC) problem, which hinders accurate retrieval and generation. In this work, we propose RagVL, a novel framework with knowledge-enhanced reranking and noise-injected training, to address these limitations. We instruction-tune the MLLM with a simple yet effective instruction template to induce its ranking ability and serve it as a reranker to precisely filter the top-k retrieved images. For generation, we inject visual noise during training at the data and token levels to enhance the generator's robustness. Extensive experiments on the subsets of two datasets that require retrieving and reasoning over images to answer a given query verify the effectiveness of our method. Code and models are available at https://github.com/IDEA-FinAI/RagVL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
