arXiv 2408.106882025-06-12cs.CV

TDS-CLIP: 用于高效视频动作识别的时序差分子网络

TDS-CLIP: Temporal Difference Side Network for Efficient VideoAction Recognition

School of Electrical and Electronic Engineering, Shandong University of Technology（山东理工大学电子与电气工程学院）；School of Control Scienceand Engineering, Shandong University（山东大学控制科学与工程学院）；School of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；Pillar of Information Systems Technology and Design, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计支柱）

Bin Wang, Wentong Li, Wenqian Wang, Mingliang Gao, Runmin Cong, Wei Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有视频动作识别预训练知识迁移方法忽略模型自身时序建模能力的问题，提出含TD-Adapter和SME-Adapter的TDS-CLIP时序差分子网络，在多基准数据集上取得有竞争力的性能。

摘要：

近年来，大规模预训练视觉语言模型（如CLIP）凭借其强大的表征能力受到广泛关注。这启发研究人员通过侧网络等方式将这些大型预训练模型的知识迁移到其他任务特定模型（如视频动作识别（VAR）模型）中，以提升参数高效微调（PEFT）的效率。然而，当前视频动作识别领域的迁移方法往往以最低成本将冻结的大型预训练模型知识直接迁移到动作识别网络，而非挖掘动作识别模型自身的时序建模能力。为此，本文提出一种新型内存高效的时序差分子网络（TDS-CLIP），在知识迁移与时序建模间取得平衡，且无需在冻结参数模型中进行反向传播。具体而言，我们引入时序差分适配器（TD-Adapter），可有效捕捉运动特征中的局部时序差异，增强模型的全局时序建模能力。此外，我们设计了侧运动增强适配器（SME-Adapter），引导所提侧网络高效学习视频中的丰富运动信息，从而提升侧网络捕捉和学习运动信息的能力。我们在三个基准数据集（包括Something-Something V1&V2以及Kinetics-400）上开展了大量实验，结果表明该方法在视频动作识别任务中取得了具有竞争力的性能。

英文摘要：

Recently, large-scale pre-trained vision-language models (e.g., CLIP), have garnered significant attention thanks to their powerful representative capabilities. This inspires researchers in transferring the knowledge from these large pre-trained models to other task-specific models, e.g., Video Action Recognition (VAR) models, via particularly leveraging side networks to enhance the efficiency of parameter-efficient fine-tuning (PEFT). However, current transferring approaches in VAR tend to directly transfer the frozen knowledge from large pre-trained models to action recognition networks with minimal cost, instead of exploiting the temporal modeling capabilities of the action recognition models themselves. Therefore, in this paper, we propose a novel memory-efficient Temporal Difference Side Network (TDS-CLIP) to balance knowledge transferring and temporal modeling, avoiding backpropagation in frozen parameter models. Specifically, we introduce a Temporal Difference Adapter (TD-Adapter), which can effectively capture local temporal differences in motion features to strengthen the model's global temporal modeling capabilities. Furthermore, we designed a Side Motion Enhancement Adapter (SME-Adapter) to guide the proposed side network in efficiently learning the rich motion information in videos, thereby improving the side network's ability to capture and learn motion information. Extensive experiments are conducted on three benchmark datasets, including Something-Something V1&V2, and Kinetics-400. Experimental results show that our method achieves competitive performance in video action recognition tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。