利用外部知识重新思考多模态大语言模型的视觉提示
Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge
AI总结：
本文提出将专门视觉模型得到的细粒度外部知识直接编码为空间视觉提示并注入 MLLM，在九个基准上提升 LLaVA、Mipha 等模型的细粒度视觉理解能力。
摘要：
近年来，多模态大语言模型（MLLMs）通过在海量高质量图文数据集上训练取得了显著进展，使其通常能够很好地理解图像。然而，文本本身难以显式传达细粒度或空间密集信息（如掩码），这给 MLLMs 带来挑战，限制了它们回答需要理解详细或局部视觉元素的问题的能力。受检索增强生成（RAG）概念启发，本文提出一种新的视觉提示方法，将从专门视觉模型（例如实例分割/OCR 模型）获取的细粒度外部知识整合到 MLLMs 中。这是提升 MLLM 性能的一个有前景但尚未充分探索的方向。我们的方法不同于同期工作：后者将外部知识转换为额外文本提示，迫使模型间接学习视觉内容与文本坐标之间的对应关系。相反，我们提出将细粒度知识信息直接嵌入空间嵌入图，作为视觉提示。该设计可轻松集成到 LLaVA 和 Mipha 等多种 MLLMs 中，显著提升其视觉理解性能。通过严格实验，我们证明该方法可在九个基准上增强 MLLM 性能，提升其细粒度上下文感知能力。
英文摘要：
In recent years, multimodal large language models (MLLMs) have made significant strides by training on vast high-quality image-text datasets, enabling them to generally understand images well. However, the inherent difficulty in explicitly conveying fine-grained or spatially dense information in text, such as masks, poses a challenge for MLLMs, limiting their ability to answer questions requiring an understanding of detailed or localized visual elements. Drawing inspiration from the Retrieval-Augmented Generation (RAG) concept, this paper proposes a new visual prompt approach to integrate fine-grained external knowledge, gleaned from specialized vision models (e.g., instance segmentation/OCR models), into MLLMs. This is a promising yet underexplored direction for enhancing MLLMs' performance. Our approach diverges from concurrent works, which transform external knowledge into additional text prompts, necessitating the model to indirectly learn the correspondence between visual content and text coordinates. Instead, we propose embedding fine-grained knowledge information directly into a spatial embedding map as a visual prompt. This design can be effortlessly incorporated into various MLLMs, such as LLaVA and Mipha, considerably improving their visual understanding performance. Through rigorous experiments, we demonstrate that our method can enhance MLLM performance across nine benchmarks, amplifying their fine-grained context-aware capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
