AnimalMotionCLIP:将运动嵌入CLIP用于动物行为分析
AnimalMotionCLIP: Embedding motion in CLIP for Animal Behavior Analysis
AI总结:
本文提出AnimalMotionCLIP,通过在CLIP框架中交错视频帧与光流信息并设计密集、半密集和稀疏分类器聚合时序建模方案,有效解决了动物行为识别中的运动信息整合与时序建模挑战,在Animal Kingdom数据集上取得优于SOTA的性能。
摘要:
近期,将深度学习技术应用于动物行为识别的兴趣激增,特别是利用CLIP等预训练视觉语言模型,因为它们在各种下游任务中具有卓越的泛化能力。然而,将这些模型适配到动物行为识别的特定领域面临两大挑战:整合运动信息和设计有效的时序建模方案。在本文中,我们提出AnimalMotionCLIP来解决这些挑战,方法是在CLIP框架中交错视频帧和光流信息。此外,还提出并比较了使用分类器聚合的几种时序建模方案:密集、半密集和稀疏。结果表明,精细的时序动作可以被正确识别,这在动物行为分析中至关重要。在Animal Kingdom数据集上的实验表明,AnimalMotionCLIP与最先进的方法相比取得了更优的性能。
英文摘要:
Recently, there has been a surge of interest in applying deep learning techniques to animal behavior recognition, particularly leveraging pre-trained visual language models, such as CLIP, due to their remarkable generalization capacity across various downstream tasks. However, adapting these models to the specific domain of animal behavior recognition presents two significant challenges: integrating motion information and devising an effective temporal modeling scheme. In this paper, we propose AnimalMotionCLIP to address these challenges by interleaving video frames and optical flow information in the CLIP framework. Additionally, several temporal modeling schemes using an aggregation of classifiers are proposed and compared: dense, semi dense, and sparse. As a result, fine temporal actions can be correctly recognized, which is of vital importance in animal behavior analysis. Experiments on the Animal Kingdom dataset demonstrate that AnimalMotionCLIP achieves superior performance compared to state-of-the-art approaches.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。