This paper presents a comprehensive exploration of the phenomenon of data redundancy in video understanding, with the aim to improve computational efficiency. Our investigation commences with an examination of spatial redundancy, which refers to the observation that the most informative region in each video frame usually corresponds to a small image patch, whose shape, size and location shift smoothly across frames. Motivated by this phenomenon, we formulate the patch localization problem as a dynamic decision task, and introduce a spatially adaptive video recognition approach, termed AdaFocus. In specific, a lightweight encoder is first employed to quickly process the full video sequence, whose features are then utilized by a policy network to identify the most task-relevant regions. Subsequently, the selected patches are inferred by a high-capacity deep network for the final prediction. The full model can be trained in end-to-end conveniently. Furthermore, AdaFocus can be extended by further considering temporal and sample-wise redundancies, i.e., allocating the majority of computation to the most task-relevant frames, and minimizing the computation spent on relatively "easier" videos. Our resulting approach, Uni-AdaFocus, establishes a comprehensive framework that seamlessly integrates spatial, temporal, and sample-wise dynamic computation, while it preserves the merits of AdaFocus in terms of efficient end-to-end training and hardware friendliness. In addition, Uni-AdaFocus is general and flexible as it is compatible with off-the-shelf efficient backbones (e.g., TSM and X3D), which can be readily deployed as our feature extractor, yielding a significantly improved computational efficiency. Empirically, extensive experiments based on seven benchmark datasets and three application scenarios substantiate that Uni-AdaFocus is considerably more efficient than the competitive baselines.
AdaFocusV3: On Unified Spatial-temporal Dynamic Video Recognition
Tsinghua University(清华大学);University of Oregon(俄勒冈大学);Picsart AI Research (PAIR)(Picsart人工智能研究院(PAIR));Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院(BAAI))
Unifying Graph Embedding Features with Graph Convolutional Networks for Skeleton-based Action Recognition
University of California, Merced(加州大学默塞德分校);Bishop’s University(毕索大学);The Education University of Hong Kong(香港教育大学);City University of Hong Kong(香港城市大学);Hefei University of Technology(合肥工业大学)
Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabularies
School of Cyber Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院,南京,中国);School of Computer Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院,南京,中国)
A Unified Deep Framework for Joint 3D Pose Estimation and Action Recognition from a Single RGB Camera
Cerema Research Center(Cerema研究中心);Aparnix;Cortexica Vision Systems Ltd.(Cortexica视觉系统有限公司);Vingroup Big Data Institute (VinBDI)(Vingroup大数据研究所(VinBDI))
HSEmotion Team at ABAW-10 Competition: Facial Expression Recognition, Valence-Arousal Estimation, Action Unit Detection and Fine-Grained Violence Classification
GenRec: Unifying Video Generation and Recognition with Diffusion Models
Fudan University(复旦大学);University of Maryland(马里兰大学);Shanghai Key Lab of Intell. Info. Processing(上海智能信息处理重点实验室);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心)
本文提出首个基于 Stable Video Diffusion 的统一框架 GenRec,通过随机帧条件化过程联合优化视频生成与识别,在多个数据集上取得优异性能,并在输入信息有限时表现出色。
本文介绍了我们在2020年野生情绪识别(Emotion Recognition in the Wild,EMOTIW)挑战赛中为音频-视频组情绪识别子任务提出的单峰隐私安全和非个人建议。这个子挑战旨在将野生视频分为三类:正面,中性和负面。最近的深度学习模型在分析人与人之间的互动、预测人类行为和情感评估方面取得了巨大的进步。尽管如此,它们的性能来自于基于个人的分析,这意味着对各个检测的分数进行总结和平均,这不可避免地会导致一些隐私问题。在这项研究中,我们研究了一种简单的方法,该方法能够从整个图像中捕获全局情绪,而无需使用人脸或姿态检测或任何基于个人的特征作为输入。所提出的方法混合了国家的最先进的和专用的合成语料库作为训练源。通过深入探索用于组级情感识别的神经网络架构,我们构建了一个基于VGG的模型,在VGAF测试集上实现了59.13%的准确率(挑战的第11位)。考虑到分析是单峰的,仅基于全局特征,并且性能是在真实世界的数据集上进行评估的,这些结果是有希望的,让我们设想将该模型扩展到多模态的课堂氛围评估,这是我们的最终目标应用。