arXiv 2401.009012024-03-30cs.CV

Video-GroundingDINO：迈向开放词汇的时空视频定位

Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding

Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；University of California, Merced（加州大学默塞德分校）；Google Research（谷歌研究院）；Linköping University（林雪平大学）

Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Video-GroundingDINO，通过利用基础空间定位模型的预训练表示，实现开放词汇时空视频定位，在多个闭集和开放词汇基准上超越现有最先进方法。

摘要：

视频定位旨在根据输入文本查询在视频中定位相应的时空片段。本文通过引入开放词汇时空视频定位任务，解决了当前视频定位方法中的一个关键局限。与因训练数据有限和预定义词汇而难以应对开放词汇场景的普遍闭集方法不同，我们的模型利用基础空间定位模型的预训练表示。这使其能够有效弥合自然语言与多样化视觉内容之间的语义差距，在闭集和开放词汇场景下均取得优异性能。我们的贡献包括一种新颖的时空视频定位模型，在多个数据集的闭集评估中超越了最先进结果，并在开放词汇场景中展现出卓越性能。值得注意的是，所提模型在VidSTG（陈述式和疑问式）和HC-STVG（V1和V2）数据集的闭集设置中优于最先进方法。此外，在HC-STVG V1和YouCook-Interactions的开放词汇评估中，我们的模型分别以4.88 m_vIoU和1.83%的准确率超越了近期最佳模型，证明了其在处理多样化语言和视觉概念以改进视频理解方面的有效性。我们的代码将公开发布。

英文摘要：

Video grounding aims to localize a spatio-temporal section in a video corresponding to an input text query. This paper addresses a critical limitation in current video grounding methodologies by introducing an Open-Vocabulary Spatio-Temporal Video Grounding task. Unlike prevalent closed-set approaches that struggle with open-vocabulary scenarios due to limited training data and predefined vocabularies, our model leverages pre-trained representations from foundational spatial grounding models. This empowers it to effectively bridge the semantic gap between natural language and diverse visual content, achieving strong performance in closed-set and open-vocabulary settings. Our contributions include a novel spatio-temporal video grounding model, surpassing state-of-the-art results in closed-set evaluations on multiple datasets and demonstrating superior performance in open-vocabulary scenarios. Notably, the proposed model outperforms state-of-the-art methods in closed-set settings on VidSTG (Declarative and Interrogative) and HC-STVG (V1 and V2) datasets. Furthermore, in open-vocabulary evaluations on HC-STVG V1 and YouCook-Interactions, our model surpasses the recent best-performing models by $4.88$ m_vIoU and $1.83\%$ accuracy, demonstrating its efficacy in handling diverse linguistic and visual concepts for improved video understanding. Our codes will be publicly released.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。