Kronecker Mask与解释性提示是语言-动作视频学习器
Kronecker Mask and Interpretive Prompts are Language-Action Video Learners
AI总结：
针对CLIP适配视频领域的问题，提出CLAVER模型，通过Kronecker掩码注意力优化视觉分支的时空建模，结合大语言模型生成的解释性提示增强文本分支的动词理解，实验验证了方法的优越性与通用性。
摘要：
对比语言-图像预训练（CLIP）显著推动了基于图像的视觉学习。随后出现了一个紧迫的课题：如何有效将CLIP适配到视频领域？近期研究聚焦于调整CLIP的文本分支或视觉分支以进行动作识别。然而，我们认为两个分支的适配都至关重要。本文提出CLAVER：一种对比语言-动作视频学习器，旨在将CLIP的焦点从静态视觉对象与具体名词的对齐转移到动态动作行为与抽象动词的对齐。具体而言，我们引入一种新颖的Kronecker掩码注意力用于时间建模。我们定制的Kronecker掩码具有三个优势：1）扩展每个token的时间感受野；2）作为有效的时空异质性归纳偏置，缓解时空同质化问题；3）可无缝嵌入基于Transformer的模型。关于文本分支，我们利用大型语言模型生成多样化、句子级且语义丰富的动作解释性提示，将模型焦点转向动词理解。在各种基准和学习场景下的广泛实验证明了我们方法的优越性和通用性。
英文摘要：
Contrastive language-image pretraining (CLIP) has significantly advanced image-based vision learning. A pressing topic subsequently arises: how can we effectively adapt CLIP to the video domain? Recent studies have focused on adjusting either the textual or visual branch of CLIP for action recognition. However, we argue that adaptations of both branches are crucial. In this paper, we propose \textbf{CLAVER}: a \textbf{C}ontrastive \textbf{L}anguage-\textbf{A}ction \textbf{V}ideo Learn\textbf{er}, designed to shift CLIP's focus from the alignment of static visual objects and concrete nouns to the alignment of dynamic action behaviors and abstract verbs. Specifically, we introduce a novel Kronecker mask attention for temporal modeling. Our tailored Kronecker mask offers three benefits 1) it expands the temporal receptive field for each token, 2) it serves as an effective spatiotemporal heterogeneity inductive bias, mitigating the issue of spatiotemporal homogenization, and 3) it can be seamlessly plugged into transformer-based models. Regarding the textual branch, we leverage large language models to generate diverse, sentence-level and semantically rich interpretive prompts of actions, which shift the model's focus towards the verb comprehension. Extensive experiments on various benchmarks and learning scenarios demonstrate the superiority and generality of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
