arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.10596cs.CV
收藏

基于现成图文特征的开放词汇时间动作检测

Open-Vocabulary Temporal Action Detection with Off-the-Shelf Image-Text Features

Google Research(谷歌研究院)

Vivek Rathod, Bryan Seybold, Sudheendra Vijayanarasimhan, Austin Myers, Xiuye Gu, Vighnesh Birodkar, David A. Ross

摘要:

在未修剪的视频中检测动作不应该局限于一个小的、封闭的类集合。我们提出了一种简单而有效的策略,利用预训练的图像-文本共嵌入进行开放词汇的时间动作检测。尽管是在静态图像而不是视频上训练的,但我们表明,图像-文本共嵌入使开放词汇表的性能能够与全监督模型竞争。我们表明,可以通过将图像-文本特征与编码局部运动的特征(如基于光流的特征)或其他模态(如音频)相结合来进一步提高性能。此外,我们提出了一个更合理的开放式词汇评价设置的ActivityNet数据集,其中的类别划分是基于相似性,而不是随机分配。

英文摘要:

Detecting actions in untrimmed videos should not be limited to a small, closed set of classes. We present a simple, yet effective strategy for open-vocabulary temporal action detection utilizing pretrained image-text co-embeddings. Despite being trained on static images rather than videos, we show that image-text co-embeddings enable openvocabulary performance competitive with fully-supervised models. We show that the performance can be further improved by ensembling the image-text features with features encoding local motion, like optical flow based features, or other modalities, like audio. In addition, we propose a more reasonable open-vocabulary evaluation setting for the ActivityNet data set, where the category splits are based on similarity rather than random assignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑