arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.13465cs.CVcs.AIcs.LG
收藏

AdaFocusV3:时空统一的动态视频识别

AdaFocusV3: On Unified Spatial-temporal Dynamic Video Recognition

Tsinghua University(清华大学);University of Oregon(俄勒冈大学);Picsart AI Research (PAIR)(Picsart人工智能研究院(PAIR));Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院(BAAI))

Yulin Wang, Yang Yue, Xinhong Xu, Ali Hassani, Victor Kulikov, Nikita Orlov, Shiji Song, Humphrey Shi, Gao Huang

摘要:

最近的研究表明,减少时间和空间冗余都是有效的视频识别方法,例如,将大部分计算分配给与任务相关的帧子集或每个帧的最有价值的图像区域。然而,在大多数现有的作品中,任何一种类型的冗余通常与另一个缺席建模。本文在最近提出的AdaFocusV 2算法的基础上,探讨了时空动态计算的统一形式,提出了一种改进的AdaFocusV 3框架。我们的方法通过仅在一些小但信息丰富的3D视频立方体上激活昂贵的高容量网络来降低计算成本。这些立方体是从由帧高度、宽度和视频持续时间形成的空间中裁剪出来的,而它们的位置是在每个样本的基础上用一个轻量级的策略网络自适应地确定的。在测试时,动态配置每个视频对应的立方体的数量,即,顺序地处理视频立方体,直到产生足够可靠的预测。值得注意的是,AdaFocusV 3可以通过用深度特征的插值来近似不可微的裁剪操作来有效地训练。六个基准数据集的广泛经验结果(即,ActivityNet、FCVID、Mini-Kinetics、Something-Something V1&V2和Diving 48)表明,我们的模型比竞争对手的基线效率高得多。

英文摘要:

Recent research has revealed that reducing the temporal and spatial redundancy are both effective approaches towards efficient video recognition, e.g., allocating the majority of computation to a task-relevant subset of frames or the most valuable image regions of each frame. However, in most existing works, either type of redundancy is typically modeled with another absent. This paper explores the unified formulation of spatial-temporal dynamic computation on top of the recently proposed AdaFocusV2 algorithm, contributing to an improved AdaFocusV3 framework. Our method reduces the computational cost by activating the expensive high-capacity network only on some small but informative 3D video cubes. These cubes are cropped from the space formed by frame height, width, and video duration, while their locations are adaptively determined with a light-weighted policy network on a per-sample basis. At test time, the number of the cubes corresponding to each video is dynamically configured, i.e., video cubes are processed sequentially until a sufficiently reliable prediction is produced. Notably, AdaFocusV3 can be effectively trained by approximating the non-differentiable cropping operation with the interpolation of deep features. Extensive empirical results on six benchmark datasets (i.e., ActivityNet, FCVID, Mini-Kinetics, Something-Something V1&V2 and Diving48) demonstrate that our model is considerably more efficient than competitive baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑