元个性化视觉语言模型以在视频中查找命名实例
Meta-Personalizing Vision-Language Models to Find Named Instances in Video
AI总结：
本文提出元个性化视觉语言模型方法，通过扩展词嵌入并在无监督下利用转录和相似性自动识别视频中的命名实例，并在新基准This-Is-My及DeepFashion2上取得15%相对提升。
摘要：
大规模视觉语言模型（VLM）在语言引导的搜索应用中展现了令人印象深刻的结果。虽然这些模型支持类别级查询，但在视频中针对特定对象实例（如“我的狗Biscuit”）出现的时刻进行个性化搜索时，它们目前仍存在困难。我们提出以下三项贡献来解决这一问题。首先，我们描述了一种对预训练VLM进行元个性化的方法，即学习如何在测试时学习个性化VLM以在视频中搜索。我们的方法通过学习针对每个实例的新颖词嵌入来扩展VLM的令牌词汇表。为了仅捕获实例特定的特征，我们将每个实例嵌入表示为共享的和学习的全局类别特征的组合。其次，我们提出在没有显式人工监督的情况下学习这种个性化。我们的方法使用转录文本和VLM嵌入空间中的视觉语言相似性自动识别视频中命名视觉实例的时刻。最后，我们引入了This-Is-My，一个个人视频实例检索基准。我们在This-Is-My和DeepFashion2上评估了我们的方法，并表明在后者数据集上我们相对于现有技术获得了15%的相对改进。
英文摘要：
Large-scale vision-language models (VLM) have shown impressive results for language-guided search applications. While these models allow category-level queries, they currently struggle with personalized searches for moments in a video where a specific object instance such as ``My dog Biscuit'' appears. We present the following three contributions to address this problem. First, we describe a method to meta-personalize a pre-trained VLM, i.e., learning how to learn to personalize a VLM at test time to search in video. Our method extends the VLM's token vocabulary by learning novel word embeddings specific to each instance. To capture only instance-specific features, we represent each instance embedding as a combination of shared and learned global category features. Second, we propose to learn such personalization without explicit human supervision. Our approach automatically identifies moments of named visual instances in video using transcripts and vision-language similarity in the VLM's embedding space. Finally, we introduce This-Is-My, a personal video instance retrieval benchmark. We evaluate our approach on This-Is-My and DeepFashion2 and show that we obtain a 15% relative improvement over the state of the art on the latter dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
