Open-VCLIP：通过插值权重优化将CLIP转变为开放词汇视频模型
Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization
AI总结：
本文提出Open-VCLIP，通过插值权重优化将CLIP改造为零样本视频分类器，在UCF、HMDB和Kinetics-600上显著超越现有方法。
摘要：
对比语言-图像预训练（CLIP）在图像理解的零样本学习能力上表现优异，但针对CLIP在零样本视频识别方面的研究仍然有限。我们提出Open-VCLIP，一种简单而有效的方法，将CLIP转变为强大的零样本视频分类器，能够在测试时识别未见过的动作和事件。我们的框架以最小的修改扩展CLIP，建模视频中的时空关系，使其成为专门的视频分类器，同时力求泛化能力。我们从形式上证明，训练Open-VCLIP等价于在零历史数据条件下进行持续学习。为解决这一问题，我们提出插值权重优化（Interpolated Weight Optimization），在训练和测试阶段均利用权重插值的优势。我们在三个流行且具有挑战性的动作识别数据集上，按照多种零样本评估协议评估了我们的方法，并证明我们的方法以明显优势超越现有最先进方法。特别是，我们在UCF、HMDB和Kinetics-600上分别取得了87.9%、58.3%和81.1%的零样本准确率，分别比现有最先进方法高出8.3%、7.8%和12.2%。代码已发布于https://github.com/wengzejia1/Open-VCLIP。
英文摘要：
Contrastive Language-Image Pretraining (CLIP) has demonstrated impressive zero-shot learning abilities for image understanding, yet limited effort has been made to investigate CLIP for zero-shot video recognition. We introduce Open-VCLIP, a simple yet effective approach that transforms CLIP into a strong zero-shot video classifier that can recognize unseen actions and events at test time. Our framework extends CLIP with minimal modifications to model spatial-temporal relationships in videos, making it a specialized video classifier, while striving for generalization. We formally show that training an Open-VCLIP is equivalent to continual learning with zero historical data. To address this problem, we propose Interpolated Weight Optimization, which utilizes the benefit of weight interpolation in both training and test time. We evaluate our method on three popular and challenging action recognition datasets following various zero-shot evaluation protocols and we demonstrate our approach outperforms state-of-the-art methods by clear margins. In particular, we achieve 87.9%, 58.3%, 81.1% zero-shot accuracy on UCF, HMDB and Kinetics-600 respectively, outperforming state-of-the-art methods by 8.3%, 7.8% and 12.2%. Code is released at https://github.com/wengzejia1/Open-VCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
