清晰可见，自信推理：用于视觉语言模型盲点的即插即用修复方法
Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness
AI总结：
本文提出一种无需微调的即插即用模块，通过细化视觉标记和丰富文本提示，提升VLM对罕见物体的推理能力。
摘要：
视觉语言模型（VLMs）在广泛的视觉理解上取得了显著成功，但它们在处理罕见物体的以对象为中心的推理时仍面临挑战，因为此类实例在预训练数据中较少。尽管先前的努力通过检索额外数据或引入更强的视觉编码器来缓解这一问题，但这些方法在微调VLMs时仍然计算密集，并未充分利用原始训练数据。在本文中，我们引入了一个高效的即插即用模块，通过细化视觉标记和丰富输入文本提示，显著提高了VLMs对罕见物体的推理能力，而无需微调VLMs。具体来说，我们通过利用视觉基础模型的先验知识和同义词增强的文本描述，学习多模态类别嵌入，以补偿有限的训练示例。这些嵌入通过一个轻量级的注意力增强模块来细化VLMs中的视觉标记，从而改进细粒度物体细节。此外，我们使用学习到的嵌入作为对象感知检测器，生成信息性提示，这些提示被注入到文本提示中，以帮助引导VLM的注意力指向相关图像区域。在两个基准测试上的实验显示，预训练VLMs在罕见物体识别和推理方面实现了稳定且显著的提升。进一步的分析揭示了我们的方法如何加强VLM对罕见物体的关注和推理能力。
英文摘要：
Vision language models (VLMs) have achieved remarkable success in broad visual understanding, yet they remain challenged by object-centric reasoning on rare objects due to the scarcity of such instances in pretraining data. While prior efforts alleviate this issue by retrieving additional data or introducing stronger vision encoders, these methods are still computationally intensive during finetuning VLMs and don't fully exploit the original training data. In this paper, we introduce an efficient plug-and-play module that substantially improves VLMs' reasoning over rare objects by refining visual tokens and enriching input text prompts, without VLMs finetuning. Specifically, we propose to learn multi-modal class embeddings for rare objects by leveraging prior knowledge from vision foundation models and synonym-augmented text descriptions, compensating for limited training examples. These embeddings refine the visual tokens in VLMs through a lightweight attention-based enhancement module that improves fine-grained object details. In addition, we use the learned embeddings as object-aware detectors to generate informative hints, which are injected into the text prompts to help guide the VLM's attention toward relevant image regions. Experiments on two benchmarks show consistent and substantial gains for pretrained VLMs in rare object recognition and reasoning. Further analysis reveals how our method strengthens the VLM's ability to focus on and reason about rare objects.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
