arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.10834cs.CVcs.AIcs.LG
收藏

Visual RAG:在不进行微调的情况下扩展MLLM的视觉知识

Visual RAG: Expanding MLLM visual knowledge without fine-tuning

University of Milano-Bicocca(米兰-比科卡大学)

Mirco Bonomo, Simone Bianco

AI总结:

本文提出Visual RAG,结合检索机制与上下文学习,无需微调即可动态扩展MLLM视觉知识,在八个数据集上以约23%的示例量达到或超过多样本ICL的准确率(平均+2%)。

摘要:

多模态大语言模型(MLLMs)在需要跨视觉和文本模态进行推理的计算机视觉任务中取得了显著性能,但其能力受限于预训练数据,需要大量微调才能更新。近期研究探索了利用上下文学习(ICL)通过提供一组示范示例作为上下文来增强MLLMs在多个任务上的性能,表明与少样本ICL相比,多样本ICL带来了显著改进。然而,对大量示范示例的依赖以及MLLMs有限的上下文窗口构成了重大障碍。本文旨在通过引入一种新颖方法Visual RAG来应对这些挑战,该方法将MLLMs从上下文中学习的能力与检索机制协同结合。该方法的关键在于通过仅选择与查询最相关的示范示例来增强MLLM知识,促使其通过类比进行学习。这样,依靠推理期间动态提供的新信息,所得系统不局限于从训练数据中提取的知识,而是可以在无需微调的情况下快速且轻松地更新。此外,这大大降低了改进模型图像分类性能的计算成本,并将模型知识扩展到其未训练过的新视觉领域和任务。在涵盖多个领域和图像分类任务的最新技术的八个不同数据集上进行的大量实验表明,与最近的最先进方法(即多样本ICL)相比,所提出的Visual RAG能够获得非常接近甚至更高的准确率(平均约+2%的提升),同时使用的示范示例集要小得多(平均仅约23%)。

英文摘要:

Multimodal Large Language Models (MLLMs) have achieved notable performance in computer vision tasks that require reasoning across visual and textual modalities, yet their capabilities are limited to their pre-trained data, requiring extensive fine-tuning for updates. Recent researches have explored the use of In-Context Learning (ICL) to overcome these challenges by providing a set of demonstrating examples as context to augment MLLMs performance in several tasks, showing that many-shot ICL leads to substantial improvements compared to few-shot ICL. However, the reliance on numerous demonstrating examples and the limited MLLMs context windows presents significant obstacles. This paper aims to address these challenges by introducing a novel approach, Visual RAG, that synergically combines the MLLMs capability to learn from the context, with a retrieval mechanism. The crux of this approach is to ensure to augment the MLLM knowledge by selecting only the most relevant demonstrating examples for the query, pushing it to learn by analogy. In this way, relying on the new information provided dynamically during inference time, the resulting system is not limited to the knowledge extracted from the training data, but can be updated rapidly and easily without fine-tuning. Furthermore, this greatly reduces the computational costs for improving the model image classification performance, and augments the model knowledge to new visual domains and tasks it was not trained for. Extensive experiments on eight different datasets in the state of the art spanning several domains and image classification tasks show that the proposed Visual RAG, compared to the most recent state of the art (i.e., many-shot ICL), is able to obtain an accuracy that is very close or even higher (approx. +2% improvement on average) while using a much smaller set of demonstrating examples (approx. only 23% on average).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑