通过上下文示例教会视觉语言模型定位特定对象
Teaching VLMs to Localize Specific Objects from In-context Examples
AI总结：
本文首次研究视觉语言模型的少样本个性化定位任务，提出基于视频目标跟踪数据微调并结合伪名称正则化的数据驱动方法，显著提升7B至72B模型的上下文定位能力而不损害泛化性。
摘要：
视觉语言模型（VLMs）在多种视觉任务中展现了卓越的能力，包括图像识别、视频理解和视觉问答（VQA），当针对这些任务进行显式训练时表现尤为突出。尽管取得了这些进展，我们发现当前的视觉语言模型（包括专有的GPT-4o）缺乏一种基本的认知能力：通过考虑上下文来学习定位场景中的特定对象。在这项工作中，我们专注于少样本个性化定位任务，即给定模型一小组带标注的图像（上下文示例）——每张图像包含一个类别标签和边界框——并要求其在查询图像中定位相同类型的对象。个性化定位在多个相关对象可能响应同一文本或对象难以用文字描述而产生歧义的情况下尤为重要。为了激发模型的个性化定位能力，我们提出了一种以数据为中心的解决方案，利用从视频对象跟踪数据集中精心筛选的数据对模型进行微调。通过利用跨多个镜头跟踪同一对象的帧序列，我们模拟了促进上下文感知的指令调优对话。为了强化这一点，我们引入了一种新颖的正则化技术，用伪名称替换对象标签，确保模型依赖视觉上下文而非先验知识。我们的方法显著提升了近期规模从7B到72B的视觉语言模型的少样本定位性能，同时不牺牲泛化能力，这一点在多个专门用于评估个性化定位能力的基准测试上得到了验证。这项工作是首次探索并基准测试视觉语言模型的个性化少样本定位——揭示了当前视觉语言模型的关键弱点，并为未来上下文驱动的视觉语言应用研究奠定了基础。
英文摘要：
Vision-Language Models (VLMs) have shown remarkable capabilities across diverse visual tasks, including image recognition, video understanding, and Visual Question Answering (VQA) when explicitly trained for these tasks. Despite these advances, we find that present-day VLMs (including the proprietary GPT-4o) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context. In this work, we focus on the task of few-shot personalized localization, where a model is given a small set of annotated images (in-context examples) -- each with a category label and bounding box -- and is tasked with localizing the same object type in a query image. Personalized localization can be particularly important in cases of ambiguity of several related objects that can respond to a text or an object that is hard to describe with words. To provoke personalized localization abilities in models, we present a data-centric solution that fine-tunes them using carefully curated data from video object tracking datasets. By leveraging sequences of frames tracking the same object across multiple shots, we simulate instruction-tuning dialogues that promote context awareness. To reinforce this, we introduce a novel regularization technique that replaces object labels with pseudo-names, ensuring the model relies on visual context rather than prior knowledge. Our method significantly enhances the few-shot localization performance of recent VLMs ranging from 7B to 72B in size, without sacrificing generalization, as demonstrated on several benchmarks tailored towards evaluating personalized localization abilities. This work is the first to explore and benchmark personalized few-shot localization for VLMs -- exposing critical weaknesses in present-day VLMs, and laying a foundation for future research in context-driven vision-language applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
