从自然脚本知识学习可迁移时空表示
Learning Transferable Spatiotemporal Representations from Natural Script Knowledge
摘要：
近年来，大规模视频数据的预训练已经成为学习可转移时空表示的常用方法。尽管取得了一些进展，但现有的方法大多限于高度策划的数据集（例如，K400），并表现出不令人满意的开箱即用的表示。我们认为，这是由于事实上，他们只捕捉像素级的知识，而不是时空语义，这阻碍了进一步的进展，视频理解。受图像-文本预训练的巨大成功的启发（例如，CLIP），我们迈出了第一步，利用语言语义来促进可转移的时空表示学习。我们引入了一个新的借口任务，转向视频成绩单排序（TVTS），排序洗牌ASR脚本出席学习视频表示。我们不依赖于描述性的字幕，而是纯粹从视频中学习，即，利用自然转录的语音知识来随时间提供有噪声但有用的语义。我们的方法强制视觉模型将随着时间的推移所发生的事情置于上下文中，以便它可以重新组织叙事文本，并可以无缝地应用于现实世界中的大规模未经策划的视频数据。我们的方法在不同的基准上展示了强大的开箱即用的时空表示，例如，通过线性探测，在SSV 2上比VideoMAE提高+13.6%。该代码可在https://github.com/TencentARC/TVTS上获得。
英文摘要：
Pre-training on large-scale video data has become a common recipe for learning transferable spatiotemporal representations in recent years. Despite some progress, existing methods are mostly limited to highly curated datasets (e.g., K400) and exhibit unsatisfactory out-of-the-box representations. We argue that it is due to the fact that they only capture pixel-level knowledge rather than spatiotemporal semantics, which hinders further progress in video understanding. Inspired by the great success of image-text pre-training (e.g., CLIP), we take the first step to exploit language semantics to boost transferable spatiotemporal representation learning. We introduce a new pretext task, Turning to Video for Transcript Sorting (TVTS), which sorts shuffled ASR scripts by attending to learned video representations. We do not rely on descriptive captions and learn purely from video, i.e., leveraging the natural transcribed speech knowledge to provide noisy but useful semantics over time. Our method enforces the vision model to contextualize what is happening over time so that it can re-organize the narrative transcripts, and can seamlessly apply to large-scale uncurated video data in the real world. Our method demonstrates strong out-of-the-box spatiotemporal representations on diverse benchmarks, e.g., +13.6% gains over VideoMAE on SSV2 via linear probing. The code is available at https://github.com/TencentARC/TVTS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
