arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2001.05691cs.CV
收藏

基于视频和文本对识别的时空特征学习

Learning Spatiotemporal Features via Video and Text Pair Discrimination

Nanjing University（南京大学）

Tianhao Li, Limin Wang

备注：
Technical Report
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

摘要：

当前的视频表示严重依赖于从手动注释的视频数据集学习，这是耗时且昂贵的获取。我们观察到视频自然伴随着丰富的文本信息，如YouTube标题和Instagram标题。在本文中，我们利用这种视觉-文本连接以有效的弱监督方式学习时空特征。我们提出了一个通用的跨模态对歧视（CPD）的框架来捕捉视频和其相关文本之间的这种相关性。具体而言，我们采用噪声对比估计来解决大量成对实例类所带来的计算问题，并设计了一个实用的课程学习策略。我们在标准视频数据集（Kinetics-210 k）和未经策划的网络视频数据集（Instagram-300 k）上训练我们的CPD模型以证明其有效性。在没有进一步微调的情况下，学习的模型在线性分类协议下获得了对Kinetics的动作分类的竞争结果。此外，我们的视觉模型提供了一个有效的初始化，以微调下游任务，这产生了显着的性能增益UCF 101和HMDB 51上的动作识别，与现有的最先进的自我监督训练方法相比。此外，我们的CPD模型产生了一个新的艺术状态的zero-shot动作识别UCF 101直接利用学习的视觉文本嵌入。该代码将在https://github.com/MCG-NJU/CPD-Video上提供。

英文摘要：

Current video representations heavily rely on learning from manually annotated video datasets which are time-consuming and expensive to acquire. We observe videos are naturally accompanied by abundant text information such as YouTube titles and Instagram captions. In this paper, we leverage this visual-textual connection to learn spatiotemporal features in an efficient weakly-supervised manner. We present a general cross-modal pair discrimination (CPD) framework to capture this correlation between a video and its associated text. Specifically, we adopt noise-contrastive estimation to tackle the computational issue imposed by the huge amount of pair instance classes and design a practical curriculum learning strategy. We train our CPD models on both standard video dataset (Kinetics-210k) and uncurated web video dataset (Instagram-300k) to demonstrate its effectiveness. Without further fine-tuning, the learnt models obtain competitive results for action classification on Kinetics under the linear classification protocol. Moreover, our visual model provides an effective initialization to fine-tune on downstream tasks, which yields a remarkable performance gain for action recognition on UCF101 and HMDB51, compared with the existing state-of-the-art self-supervised training methods. In addition, our CPD model yields a new state of the art for zero-shot action recognition on UCF101 by directly utilizing the learnt visual-textual embeddings. The code will be made available at https://github.com/MCG-NJU/CPD-Video.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑