arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.11228cs.CVcs.AIcs.LG
收藏

Uni-AdaFocus:用于视频识别的时空动态计算

Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognition

Tsinghua University(清华大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);China Mobile Research Institute(中国移动研究院)

Yulin Wang, Haoji Zhang, Yang Yue, Shiji Song, Chao Deng, Junlan Feng, Gao Huang

AI总结:

本文提出Uni-AdaFocus,一种结合空间、时间和样本级动态计算的视频识别框架,通过轻量级编码器和策略网络定位关键区域,并兼容高效骨干网络,显著提升计算效率。

摘要:

本文全面探索了视频理解中的数据冗余现象,旨在提高计算效率。我们的研究首先考察了空间冗余,即观察到每个视频帧中最具信息量的区域通常对应一个小图像块,其形状、大小和位置在帧间平滑移动。受此现象启发,我们将补丁定位问题表述为动态决策任务,并引入了一种空间自适应的视频识别方法,称为AdaFocus。具体而言,首先使用轻量级编码器快速处理整个视频序列,然后利用策略网络根据这些特征识别出最与任务相关的区域。随后,由高容量深度网络对选定的补丁进行推理以进行最终预测。整个模型可以方便地进行端到端训练。此外,AdaFocus可以通过进一步考虑时间和样本级冗余来扩展,即将大部分计算分配给最与任务相关的帧,并最小化在相对“较容易”的视频上花费的计算量。我们由此得到的方法Uni-AdaFocus建立了一个综合框架,无缝集成了空间、时间和样本级动态计算,同时保留了AdaFocus在高效端到端训练和硬件友好性方面的优点。此外,Uni-AdaFocus是通用且灵活的,因为它与现成的高效骨干网络(如TSM和X3D)兼容,这些网络可以轻松部署为我们的特征提取器,从而显著提高计算效率。实证上,基于七个基准数据集和三个应用场景的大量实验证明,Uni-AdaFocus比竞争基线方法效率高得多。

英文摘要:

This paper presents a comprehensive exploration of the phenomenon of data redundancy in video understanding, with the aim to improve computational efficiency. Our investigation commences with an examination of spatial redundancy, which refers to the observation that the most informative region in each video frame usually corresponds to a small image patch, whose shape, size and location shift smoothly across frames. Motivated by this phenomenon, we formulate the patch localization problem as a dynamic decision task, and introduce a spatially adaptive video recognition approach, termed AdaFocus. In specific, a lightweight encoder is first employed to quickly process the full video sequence, whose features are then utilized by a policy network to identify the most task-relevant regions. Subsequently, the selected patches are inferred by a high-capacity deep network for the final prediction. The full model can be trained in end-to-end conveniently. Furthermore, AdaFocus can be extended by further considering temporal and sample-wise redundancies, i.e., allocating the majority of computation to the most task-relevant frames, and minimizing the computation spent on relatively "easier" videos. Our resulting approach, Uni-AdaFocus, establishes a comprehensive framework that seamlessly integrates spatial, temporal, and sample-wise dynamic computation, while it preserves the merits of AdaFocus in terms of efficient end-to-end training and hardware friendliness. In addition, Uni-AdaFocus is general and flexible as it is compatible with off-the-shelf efficient backbones (e.g., TSM and X3D), which can be readily deployed as our feature extractor, yielding a significantly improved computational efficiency. Empirically, extensive experiments based on seven benchmark datasets and three application scenarios substantiate that Uni-AdaFocus is considerably more efficient than the competitive baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑