从图像描述学习中学习文本到视频检索
Learning text-to-video retrieval from image captioning
AI总结：
本文提出一种利用无标签视频进行文本到视频检索训练的协议，通过图像描述模型为视频帧自动生成伪标签并提供监督信号，无需人工标注即可适配目标域，在三个标准数据集上超越了CLIP零样本基线。
摘要：
我们描述了一种使用无标签视频研究文本到视频检索训练的协议，其中我们假设（i）无法访问任何视频的标签，即无法访问真实标签描述集合，但（ii）可以访问以文本形式存在的带标签图像。考虑到标注图像成本更低因此可扩展，使用图像专家模型是一个现实的场景，这与昂贵的视频标注方案形成对比。最近，像CLIP这样的零样本图像专家为视频理解任务建立了一个新的强基线。在本文中，我们利用这一进展，从两类模型中实例化图像专家：一个文本到图像检索模型用于提供初始主干网络，以及图像描述模型用于为无标签视频提供监督信号。我们展示了用图像描述自动标注视频帧可以实现文本到视频检索训练。这一过程无需任何人工标注成本即可将特征适配到目标领域，从而超越了强零样本CLIP基线。在训练期间，我们从最匹配视觉内容的多个视频帧中采样描述，并根据每个帧与各描述的相关性对帧进行评分，从而对帧表示执行时间池化。我们进行了广泛的消融研究以提供见解，并通过在三个标准数据集即ActivityNet、MSR-VTT和MSVD上的文本到视频检索中超越CLIP零样本基线，证明了这一简单框架的有效性。
英文摘要：
We describe a protocol to study text-to-video retrieval training with unlabeled videos, where we assume (i) no access to labels for any videos, i.e., no access to the set of ground-truth captions, but (ii) access to labeled images in the form of text. Using image expert models is a realistic scenario given that annotating images is cheaper therefore scalable, in contrast to expensive video labeling schemes. Recently, zero-shot image experts such as CLIP have established a new strong baseline for video understanding tasks. In this paper, we make use of this progress and instantiate the image experts from two types of models: a text-to-image retrieval model to provide an initial backbone, and image captioning models to provide supervision signal into unlabeled videos. We show that automatically labeling video frames with image captioning allows text-to-video retrieval training. This process adapts the features to the target domain at no manual annotation cost, consequently outperforming the strong zero-shot CLIP baseline. During training, we sample captions from multiple video frames that best match the visual content, and perform a temporal pooling over frame representations by scoring frames according to their relevance to each caption. We conduct extensive ablations to provide insights and demonstrate the effectiveness of this simple framework by outperforming the CLIP zero-shot baselines on text-to-video retrieval on three standard datasets, namely ActivityNet, MSR-VTT, and MSVD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
