使用多模态提示的端到端开放词汇视频视觉关系检测
End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Prompting
AI总结：
针对开放词汇视频视觉关系检测中现有方法依赖预训练轨迹检测器导致泛化能力差的问题，提出端到端开放词汇框架，包含关系感知轨迹检测器和多模态提示分类器，提升了对新类别的泛化性能。
摘要：
开放词汇视频视觉关系检测旨在突破标注类别限制，检测视频中已见和未见物体间的未见关系，扩展视频视觉关系检测的范围。现有方法通常使用在封闭数据集上训练的轨迹检测器检测物体轨迹，再将这些轨迹输入大规模预训练视觉语言模型实现开放词汇分类。这种对预训练轨迹检测器的严重依赖限制了其对新物体类别的泛化能力，导致性能下降。为应对这一挑战，我们提出将物体轨迹检测和关系分类统一到端到端开放词汇框架中。在此框架下，我们提出关系感知的开放词汇轨迹检测器，其主要包含基于查询的Transformer解码器（其中蒸馏了CLIP的视觉编码器用于逐帧开放词汇物体检测）和轨迹关联器。为在轨迹检测过程中利用关系上下文，我们将关系查询嵌入Transformer解码器，并相应设计辅助关系损失，使解码器能显式感知物体间的关系。此外，我们提出开放词汇关系分类器，利用CLIP的丰富语义知识发现新关系。为使CLIP更好适应关系分类，我们设计多模态提示方法，采用时空视觉提示获取视觉表示，采用视觉引导的语言提示处理语言输入。在VidVRD和VidOR两个公开数据集上的大量实验证明了我们框架的有效性，我们的框架还应用于更困难的跨数据集场景，进一步展示了其泛化能力。
英文摘要：
Open-vocabulary video visual relationship detection aims to expand video visual relationship detection beyond annotated categories by detecting unseen relationships between both seen and unseen objects in videos. Existing methods usually use trajectory detectors trained on closed datasets to detect object trajectories, and then feed these trajectories into large-scale pre-trained vision-language models to achieve open-vocabulary classification. Such heavy dependence on the pre-trained trajectory detectors limits their ability to generalize to novel object categories, leading to performance degradation. To address this challenge, we propose to unify object trajectory detection and relationship classification into an end-to-end open-vocabulary framework. Under this framework, we propose a relationship-aware open-vocabulary trajectory detector. It primarily consists of a query-based Transformer decoder, where the visual encoder of CLIP is distilled for frame-wise open-vocabulary object detection, and a trajectory associator. To exploit relationship context during trajectory detection, a relationship query is embedded into the Transformer decoder, and accordingly, an auxiliary relationship loss is designed to enable the decoder to perceive the relationships between objects explicitly. Moreover, we propose an open-vocabulary relationship classifier that leverages the rich semantic knowledge of CLIP to discover novel relationships. To adapt CLIP well to relationship classification, we design a multi-modal prompting method that employs spatio-temporal visual prompting for visual representation and vision-guided language prompting for language input. Extensive experiments on two public datasets, VidVRD and VidOR, demonstrate the effectiveness of our framework. Our framework is also applied to a more difficult cross-dataset scenario to further demonstrate its generalization ability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
