构建具备更优架构、优化方式与数据的开放词汇视频CLIP模型
Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Data
AI总结：
针对CLIP在零样本视频识别领域探索不足的问题，提出Open-VCLIP++框架，通过插值权重优化、大语言模型生成细粒度视频描述对齐等技术，在多个动作识别和视频检索数据集上取得优于现有方法的性能。
摘要：
尽管对比语言-图像预训练（Contrastive Language-Image Pretraining，CLIP）在零样本图像识别中已取得显著成果，但针对其在零样本视频识别领域潜力的探索仍较为有限。本文提出Open-VCLIP++，这是一个简洁且有效的框架，可将CLIP适配为性能强劲的零样本视频分类器，能够在测试阶段识别全新的动作与事件。Open-VCLIP++对CLIP进行最小程度的修改，以捕捉视频中的时空关系，从而在追求泛化能力的同时构建专用的视频分类器。我们从形式上证明，训练Open-VCLIP++等价于无历史数据的持续学习。为解决这一问题，我们提出插值权重优化（Interpolated Weight Optimization）技术，该技术在训练和测试阶段均能发挥权重插值的优势。此外，我们基于大语言模型生成细粒度的视频描述，这些详细描述进一步与视频特征对齐，助力CLIP更好地迁移到视频领域。我们在三个广泛使用的动作识别数据集上，遵循多种零样本评估协议对所提方法进行评估。结果表明，我们的方法大幅超越现有最先进技术。具体而言，我们在UCF、HMDB和Kinetics-600数据集上分别取得了88.1%、58.7%和81.2%的零样本准确率，比表现最优的对比方法分别高出8.5%、8.2%和12.3%。我们还在MSR-VTT视频-文本检索数据集上评估了该方法，其在视频到文本和文本到视频检索任务上均取得了具有竞争力的性能，同时与其他方法相比所需的微调数据大幅减少。代码已发布于https://github.com/wengzejia1/Open-VCLIP。
英文摘要：
Despite significant results achieved by Contrastive Language-Image Pretraining (CLIP) in zero-shot image recognition, limited effort has been made exploring its potential for zero-shot video recognition. This paper presents Open-VCLIP++, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing. Open-VCLIP++ minimally modifies CLIP to capture spatial-temporal relationships in videos, thereby creating a specialized video classifier while striving for generalization. We formally demonstrate that training Open-VCLIP++ is tantamount to continual learning with zero historical data. To address this problem, we introduce Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing. Furthermore, we build upon large language models to produce fine-grained video descriptions. These detailed descriptions are further aligned with video features, facilitating a better transfer of CLIP to the video domain. Our approach is evaluated on three widely used action recognition datasets, following a variety of zero-shot evaluation protocols. The results demonstrate that our method surpasses existing state-of-the-art techniques by significant margins. Specifically, we achieve zero-shot accuracy scores of 88.1%, 58.7%, and 81.2% on UCF, HMDB, and Kinetics-600 datasets respectively, outpacing the best-performing alternative methods by 8.5%, 8.2%, and 12.3%. We also evaluate our approach on the MSR-VTT video-text retrieval dataset, where it delivers competitive video-to-text and text-to-video retrieval performance, while utilizing substantially less fine-tuning data compared to other methods. Code is released at https://github.com/wengzejia1/Open-VCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
