HowTo100M：通过观看上亿个叙述视频片段学习文本-视频嵌入
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
摘要：
学习文本视频嵌入通常需要一个视频剪辑数据集，并手动提供字幕。然而，这样的数据集是昂贵的和耗时的创建，因此难以获得大规模。在这项工作中，我们建议，而不是学习这样的嵌入视频数据与现成的自然语言注释的形式自动转录的叙述。这项工作的贡献是三方面的。首先，我们介绍了HowTo 100 M：一个包含1.36亿个视频片段的大规模数据集，这些视频片段来自1.22万个描述人类执行和描述超过23 k个不同视觉任务的叙事教学网络视频。我们的数据汇集过程快速、可扩展，不需要任何额外的手动注释。其次，我们证明了在此数据上训练的文本视频嵌入可以在YouCook 2或CrossTask等教学视频数据集上获得最先进的文本到视频检索和动作本地化结果。最后，我们证明了这种嵌入可以很好地转移到其他领域：在通用Youtube视频（MSR-VTT数据集）和电影（LSMDC数据集）上进行微调，其性能优于仅在这些数据集上训练的模型。我们的数据集，代码和模型将在www.di.ens.fr/willow/research/howto100m/上公开。
英文摘要：
Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models will be publicly available at: www.di.ens.fr/willow/research/howto100m/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
