OmniCLIP：通过时空全尺度特征学习适配CLIP用于视频识别
OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learning
AI总结：
针对CLIP在视频识别中时空特征捕捉整合不足的问题，提出OmniCLIP框架，通过含并行时间适配器的时空模块和自提示生成器学习全尺度特征，在多类视频识别任务上取得优异性能。
摘要：
近期的视觉语言模型（VLM）例如CLIP在视频识别领域取得了长足进展。尽管强大的视觉骨干网络在提取空间特征方面带来了性能提升，但CLIP在捕捉和整合时空特征方面仍存在不足，而时空特征对视频识别至关重要。本文提出OmniCLIP框架，该框架通过专注于学习涵盖空间、时间以及动态时空尺度的综合特征（我们称之为全尺度特征），将CLIP适配到视频识别任务中。这一目标通过设计包含并行时间适配器（PTA）的时空模块实现，可支持高效的时间建模。此外，我们引入了自提示生成器（SPG）模块，用于捕捉动态物体的空间特征。PTA与SPG的协同作用使OmniCLIP能够识别不同帧间变化的空间信息，并评估随时间变化的物体尺度。我们在有监督视频识别、少样本视频识别和零样本识别任务上开展了大量实验。结果证明了我们方法的有效性，尤其是OmniCLIP在16样本设置下于HMDB51数据集上取得了74.30%的top-1准确率，即便使用全量训练数据，也超越了近期的MotionPrompt方法。代码可在https://github.com/XiaoBuL/OmniCLIP获取。
英文摘要：
Recent Vision-Language Models (VLMs) \textit{e.g.} CLIP have made great progress in video recognition. Despite the improvement brought by the strong visual backbone in extracting spatial features, CLIP still falls short in capturing and integrating spatial-temporal features which is essential for video recognition. In this paper, we propose OmniCLIP, a framework that adapts CLIP for video recognition by focusing on learning comprehensive features encompassing spatial, temporal, and dynamic spatial-temporal scales, which we refer to as omni-scale features. This is achieved through the design of spatial-temporal blocks that include parallel temporal adapters (PTA), enabling efficient temporal modeling. Additionally, we introduce a self-prompt generator (SPG) module to capture dynamic object spatial features. The synergy between PTA and SPG allows OmniCLIP to discern varying spatial information across frames and assess object scales over time. We have conducted extensive experiments in supervised video recognition, few-shot video recognition, and zero-shot recognition tasks. The results demonstrate the effectiveness of our method, especially with OmniCLIP achieving a top-1 accuracy of 74.30\% on HMDB51 in a 16-shot setting, surpassing the recent MotionPrompt approach even with full training data. The code is available at \url{https://github.com/XiaoBuL/OmniCLIP}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
