提示感知适配器：面向多模态大语言模型的学习自适应视觉令牌
Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models
AI总结：
本文提出提示感知适配器，根据提示动态生成视觉令牌，减轻多模态大语言模型在复杂场景中的认知负担，并在视觉问答任务中验证其有效性。
摘要：
为弥合视觉与语言模态之间的差距，多模态大语言模型（MLLMs）通常学习一个适配器，将视觉输入转换为大语言模型（LLMs）可理解的令牌。然而，大多数适配器生成一致的视觉令牌，而不考虑提示中提到的具体关注对象。由于这些适配器对图像中的每个细节分配相同的注意力，并关注整个场景，它们可能增加LLMs的认知负担，尤其是在处理复杂场景时。为缓解此问题，我们提出提示感知适配器。这些适配器被设计为具有基于提示的具体关注点动态嵌入视觉输入的能力。具体而言，提示感知适配器利用全局和局部文本特征，在粗粒度和细粒度级别上从提示中捕获最相关的视觉线索。这种方法显著增强了LLMs理解和解释视觉内容的能力。在各种视觉问答任务（如计数和位置推理）上的实验证明了提示感知适配器的有效性。
英文摘要：
To bridge the gap between vision and language modalities, Multimodal Large Language Models (MLLMs) usually learn an adapter that converts visual inputs to understandable tokens for Large Language Models (LLMs). However, most adapters generate consistent visual tokens, regardless of the specific objects of interest mentioned in the prompt. Since these adapters distribute equal attention to every detail in the image and focus on the entire scene, they may increase the cognitive load for LLMs, particularly when processing complex scenes. To alleviate this problem, we propose prompt-aware adapters. These adapters are designed with the capability to dynamically embed visual inputs based on the specific focus of the prompt. Specifically, prompt-aware adapters utilize both global and local textual features to capture the most relevant visual clues from the prompt at both coarse and fine granularity levels. This approach significantly enhances the ability of LLMs to understand and interpret visual content. Experiments on various visual question answering tasks, such as counting and position reasoning, demonstrate the effectiveness of prompt-aware adapters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
