在流动中看见：通过运动提示学习适配 CLIP 用于动作识别
Seeing in Flowing: Adapting CLIP for Action Recognition with Motion Prompts Learning
AI总结：
本文提出通过双流运动建模、动态运动提示学习和多模态通信模块适配CLIP，在少样本、零样本和闭集动作识别中取得领先性能。
摘要：
对比语言-图像预训练（CLIP）最近在“零样本”训练上表现出显著的泛化能力，并已被应用于许多下游任务。我们探索对 CLIP 进行适配，以实现一种更高效且更具泛化能力的动作识别方法。我们认为关键在于显式建模视频帧中流动的运动线索。为此，我们设计了一个双流运动建模模块，以同时捕获运动信息和空间信息。然后，利用所获得的运动线索驱动一个动态提示学习器生成运动感知提示，这些提示包含大量与人类动作相关的语义信息。此外，我们提出了一个多模态通信模块，以实现协同学习并进一步提升性能。我们在 HMDB-51、UCF-101 和 Kinetics-400 数据集上进行了大量实验。我们的方法在“少样本”和“零样本”训练上以显著优势优于大多数现有最先进方法。我们还在“闭集”训练中取得了具有竞争力的性能，且可训练参数极少、额外计算成本极低。
英文摘要：
The Contrastive Language-Image Pre-training (CLIP) has recently shown remarkable generalization on "zero-shot" training and has applied to many downstream tasks. We explore the adaptation of CLIP to achieve a more efficient and generalized action recognition method. We propose that the key lies in explicitly modeling the motion cues flowing in video frames. To that end, we design a two-stream motion modeling block to capture motion and spatial information at the same time. And then, the obtained motion cues are utilized to drive a dynamic prompts learner to generate motion-aware prompts, which contain much semantic information concerning human actions. In addition, we propose a multimodal communication block to achieve a collaborative learning and further improve the performance. We conduct extensive experiments on HMDB-51, UCF-101, and Kinetics-400 datasets. Our method outperforms most existing state-of-the-art methods by a significant margin on "few-shot" and "zero-shot" training. We also achieve competitive performance on "closed-set" training with extremely few trainable parameters and additional computational costs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
