OpenVidVRD：通过提示驱动的语义空间对齐实现开放词汇视频视觉关系检测
OpenVidVRD: Open-Vocabulary Video Visual Relation Detection via Prompt-Driven Semantic Space Alignment
AI总结：
本文提出OpenVidVRD框架，通过提示学习迁移视觉语言模型知识，结合时空细化模块和提示驱动的语义空间对齐策略，在开放词汇视频视觉关系检测任务上超越现有方法。
摘要：
视频视觉关系检测（VidVRD）任务旨在识别视频中的物体及其关系，由于视频内容的动态性、高昂的标注成本以及关系分布的长尾特性，该任务极具挑战性。视觉语言模型（VLMs）有助于探索开放词汇视觉关系检测任务，但往往忽略了不同视觉区域与其关系之间的联系。此外，由于图像与视频之间存在巨大差异，直接使用VLMs识别视频中的视觉关系面临显著困难。因此，我们提出了一种新颖的开放词汇VidVRD框架，称为OpenVidVRD，该框架通过提示学习迁移VLMs丰富的知识和强大能力以改进VidVRD任务。具体而言，我们使用VLM基于视频区域自动生成的区域描述来提取文本表示。接着，我们设计了一个时空细化模块，通过整合跨模态时空互补信息来推导视频中物体级别的关系表示。此外，我们采用了一种提示驱动的语义空间对齐策略，以充分利用VLMs的语义理解能力，从而增强OpenVidVRD的整体泛化能力。在VidVRD和VidOR公开数据集上进行的大量实验表明，所提出的模型优于现有方法。
英文摘要：
The video visual relation detection (VidVRD) task is to identify objects and their relationships in videos, which is challenging due to the dynamic content, high annotation costs, and long-tailed distribution of relations. Visual language models (VLMs) help explore open-vocabulary visual relation detection tasks, yet often overlook the connections between various visual regions and their relations. Moreover, using VLMs to directly identify visual relations in videos poses significant challenges because of the large disparity between images and videos. Therefore, we propose a novel open-vocabulary VidVRD framework, termed OpenVidVRD, which transfers VLMs' rich knowledge and powerful capabilities to improve VidVRD tasks through prompt learning. Specificall y, We use VLM to extract text representations from automatically generated region captions based on the video's regions. Next, we develop a spatiotemporal refiner module to derive object-level relationship representations in the video by integrating cross-modal spatiotemporal complementary information. Furthermore, a prompt-driven strategy to align semantic spaces is employed to harness the semantic understanding of VLMs, enhancing the overall generalization ability of OpenVidVRD. Extensive experiments conducted on the VidVRD and VidOR public datasets show that the proposed model outperforms existing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
