GRIP：面向大型多模态模型的反馈引导提示检索
GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models
AI总结：
提出GRIP，一种可学习的视觉检索框架，利用多模态模型反馈识别真正提升上下文学习性能的示例，在分类、描述和VQA任务上优于基于相似度的检索。
摘要：
上下文学习（ICL）已成为一种强大的机制，使大型语言模型（LLMs）无需微调即可适应新任务。将此概念扩展到大型多模态模型（LMMs），多模态上下文学习（M-ICL）依赖于检索相关示例（如图像、标题或问答对）来指导分类、描述和视觉问答（VQA）等任务的预测。现有方法大多基于特征空间相似性选择上下文示例，假设语义相似的样本提供最有用的上下文。然而，我们的系统分析表明，这一假设并不总是成立：视觉上相似的示例并不一定是那些最有效增强上下文学习性能的示例。为解决此问题，我们提出了上下文提示的引导检索（GRIP），一种可学习的纯视觉检索框架，利用LMMs的反馈来识别真正改善模型预测的示例。GRIP通过对比训练学习区分有益和有害的上下文示例，将检索优化到超越纯相似性。在三个多模态任务（分类、描述和VQA）上，GRIP在Qwen2.5-VL-7B上持续优于基于相似度的检索，在Idefics2-8B上的分类任务中提升最为显著。此外，我们证明了从一个开放LMM训练得到的检索器可以迁移到其他模型（包括闭源的GPT-4o和Gemini）而无需重新训练，从而实现了M-ICL的可扩展且经济高效的部署。代码将在接收后发布。
英文摘要：
In-Context Learning (ICL) has become a powerful mechanism for adapting Large Language Models (LLMs) to new tasks without fine-tuning. Extending this concept to Large Multimodal Models (LMMs), Multimodal In-Context Learning (M-ICL) relies on retrieving relevant examples, such as images, captions, or question-answer pairs, to guide predictions across tasks like classification, captioning, and visual question answering (VQA). Most existing approaches select in-context examples based on feature-space similarity, assuming that semantically similar samples provide the most useful context. However, our systematic analysis reveals that this assumption does not always hold: visually similar examples are not necessarily those that most effectively enhance in-context learning performance. To address this, we propose the Guided Retrieval of In-context Prompts (GRIP), a learnable vision-only retrieval framework that leverages feedback from LMMs to identify examples that truly improve model predictions. GRIP learns to distinguish beneficial from detrimental in-context examples through contrastive training, refining retrieval beyond pure similarity. Across three multimodal tasks, namely classification, captioning, and VQA, GRIP improves consistently over similarity-based retrieval on Qwen2.5-VL-7B, with its strongest gains in classification on Idefics2-8B. Moreover, we demonstrate that retrievers trained with feedback from one open LMM can be transferred to other models without retraining, including closed-source GPT-4o and Gemini, enabling scalable and cost-efficient deployment of M-ICL. Code will be published upon acceptance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
