arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.03307cs.CVeess.IV
收藏

Vita-CLIP:基于多模态提示的视频与文本自适应CLIP

Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting

Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学);University of Central Florida(中佛罗里达大学)

Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, Mubarak Shah

AI总结:

提出多模态提示学习方案Vita-CLIP,在冻结骨干网络下统一训练,平衡视频动作识别的监督与零样本性能,实现最先进零样本表现并保持监督竞争力。

摘要:

将CLIP等对比图像-文本预训练模型应用于视频分类因其成本效益和有竞争力的性能而备受关注。然而,该领域近期的研究面临一个权衡。微调预训练模型以获得强大的监督性能会导致零样本泛化能力低下。同样,冻结骨干网络以保留零样本能力会导致监督准确率显著下降。因此,文献中近期的研究通常为监督和零样本动作识别训练单独的模型。在本研究中,我们提出了一种多模态提示学习方案,旨在单次统一训练下平衡监督和零样本性能。我们在视觉侧的提示方法涵盖三个方面:1)全局视频级提示,用于建模数据分布;2)局部帧级提示,用于提供逐帧的判别性条件;3)摘要提示,用于提取紧凑的视频表示。此外,我们在文本侧定义了提示方案以增强文本上下文。通过此提示方案,我们能在Kinetics-600、HMDB51和UCF101上实现最先进的零样本性能,同时在监督设置中保持竞争力。通过保持预训练骨干网络冻结,我们优化了数量少得多的参数,并保留了现有的通用表示,这有助于实现强大的零样本性能。我们的代码/模型已发布于https://github.com/TalalWasim/Vita-CLIP。

英文摘要:

Adopting contrastive image-text pretrained models like CLIP towards video classification has gained attention due to its cost-effectiveness and competitive performance. However, recent works in this area face a trade-off. Finetuning the pretrained model to achieve strong supervised performance results in low zero-shot generalization. Similarly, freezing the backbone to retain zero-shot capability causes significant drop in supervised accuracy. Because of this, recent works in literature typically train separate models for supervised and zero-shot action recognition. In this work, we propose a multimodal prompt learning scheme that works to balance the supervised and zero-shot performance under a single unified training. Our prompting approach on the vision side caters for three aspects: 1) Global video-level prompts to model the data distribution; 2) Local frame-level prompts to provide per-frame discriminative conditioning; and 3) a summary prompt to extract a condensed video representation. Additionally, we define a prompting scheme on the text side to augment the textual context. Through this prompting scheme, we can achieve state-of-the-art zero-shot performance on Kinetics-600, HMDB51 and UCF101 while remaining competitive in the supervised setting. By keeping the pretrained backbone frozen, we optimize a much lower number of parameters and retain the existing general representation which helps achieve the strong zero-shot performance. Our codes/models are released at https://github.com/TalalWasim/Vita-CLIP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑