arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.00268cs.CV
收藏

用于开放词汇视频关系检测的具有运动线索的成分提示微调

Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection

Zhejiang University(浙江大学);The Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学);Singapore Management University(新加坡管理大学)

Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, Qianru Sun

摘要:

利用大规模预训练的视觉语言模型进行提示微调,可以在有限的基本类别上训练开放词汇预测,例如,目标分类和检测。在本文中,我们提出了组成提示微调与运动线索:一个扩展的提示微调模式的组成预测的视频数据。特别是,我们提出了用于开放词汇视频视觉关系检测(Open-VidVRD)的关系提示(RePro),其中传统的提示微调很容易偏向于某些主体-对象组合和运动模式。为此,RePro解决了Open-VidVRD的两个技术挑战:1)提示标记应该尊重主体和客体的两个不同语义角色,以及2)调整应该考虑主体-客体组合的不同时空运动模式。没有花里胡哨的东西,我们的RePro在两个VidVRD基准测试上实现了最先进的性能,不仅包括基本训练对象和谓词类别,还包括看不见的类别。广泛的消融也证明了建议的成分和多模式设计的提示的有效性。代码可在https://github.com/Dawn-LX/OpenVoc-VidVRD上获得。

英文摘要:

Prompt tuning with large-scale pretrained vision-language models empowers open-vocabulary predictions trained on limited base categories, e.g., object classification and detection. In this paper, we propose compositional prompt tuning with motion cues: an extended prompt tuning paradigm for compositional predictions of video data. In particular, we present Relation Prompt (RePro) for Open-vocabulary Video Visual Relation Detection (Open-VidVRD), where conventional prompt tuning is easily biased to certain subject-object combinations and motion patterns. To this end, RePro addresses the two technical challenges of Open-VidVRD: 1) the prompt tokens should respect the two different semantic roles of subject and object, and 2) the tuning should account for the diverse spatio-temporal motion patterns of the subject-object compositions. Without bells and whistles, our RePro achieves a new state-of-the-art performance on two VidVRD benchmarks of not only the base training object and predicate categories, but also the unseen ones. Extensive ablations also demonstrate the effectiveness of the proposed compositional and multi-mode design of prompts. Code is available at https://github.com/Dawn-LX/OpenVoc-VidVRD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑