面向多通道视频-语言检索的预训练对比模型快速适配研究
Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval
AI总结：
本文提出通过设计空间探索视频表示与文本融合方式，发现离散文本token结合预训练对比文本模型可在有限资源下高效适配多通道视频-语言检索，并在iVQA和How2QA上超越现有最优方法。
摘要：
多通道视频-语言检索要求模型理解来自不同通道的信息（例如视频+问题、视频+语音），以便正确地将视频与文本回答或查询关联起来。幸运的是，对比多模态模型（如CLIP）已被证明在对齐图像/视频与文本实体方面非常有效；文本对比模型（如SimCSE）因其生成判别性句子嵌入的强大能力，近年来也得到了广泛研究。然而，目前尚缺乏一种清晰的方法，能够在数据和资源有限的情况下，将这两类模型快速适配到多通道视频-语言检索任务中。本文中，我们确定了一个具有两个轴的原则性模型设计空间：如何表示视频，以及如何融合视频与文本信息。基于对近期方法的分类，我们研究了使用连续特征向量或离散文本token来表示视频的选项；在融合方法方面，我们探索了使用多模态transformer或预训练对比文本模型。我们在五个视频-语言数据集上对四种组合进行了广泛评估。我们惊讶地发现，离散文本token与预训练对比文本模型相结合能够取得最佳性能，甚至在iVQA和How2QA数据集上无需在数百万视频-文本数据上进行额外训练，就能超越当前最先进的方法。进一步分析表明，这是因为将视频表示为文本token能够捕获关键的视觉信息，而文本token天然地与经过对比预训练后成为强大检索器的文本模型对齐。所有实证分析为未来关于低成本且可升级的多模态智能研究奠定了坚实基础。
英文摘要：
Multi-channel video-language retrieval require models to understand information from different channels (e.g. video$+$question, video$+$speech) to correctly link a video with a textual response or query. Fortunately, contrastive multimodal models are shown to be highly effective at aligning entities in images/videos and text, e.g., CLIP; text contrastive models are extensively studied recently for their strong ability of producing discriminative sentence embeddings, e.g., SimCSE. However, there is not a clear way to quickly adapt these two lines to multi-channel video-language retrieval with limited data and resources. In this paper, we identify a principled model design space with two axes: how to represent videos and how to fuse video and text information. Based on categorization of recent methods, we investigate the options of representing videos using continuous feature vectors or discrete text tokens; for the fusion method, we explore the use of a multimodal transformer or a pretrained contrastive text model. We extensively evaluate the four combinations on five video-language datasets. We surprisingly find that discrete text tokens coupled with a pretrained contrastive text model yields the best performance, which can even outperform state-of-the-art on the iVQA and How2QA datasets without additional training on millions of video-text data. Further analysis shows that this is because representing videos as text tokens captures the key visual information and text tokens are naturally aligned with text models that are strong retrievers after the contrastive pretraining process. All the empirical analysis establishes a solid foundation for future research on affordable and upgradable multimodal intelligence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
