arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.10771cs.CV
收藏

EAN:用于增强动作识别的事件自适应网络

EAN: Event Adaptive Network for Enhanced Action Recognition

Shanghai Jiao Tong University(上海交通大学);Baidu(百度)

Yuan Tian, Yichao Yan, Guangtao Zhai, Guodong Guo, Zhiyong Gao

摘要:

有效地建模视频中的时空信息是动作识别的关键。为了实现这一目标,最先进的方法通常采用卷积算子和密集的交互模块,如非局部块。然而,这些方法不能准确地适应视频中的各种事件。一方面,所采用的卷积具有固定的尺度,从而与各种尺度的事件进行斗争。另一方面,密集交互建模范式只能实现次优性能,因为与动作无关的部分会为最终预测带来额外的噪声。在本文中,我们提出了一个统一的动作识别框架,通过引入以下设计来研究视频内容的动态特性。首先,在提取局部线索时,我们生成动态尺度的时空核,以适应不同的事件。其次,准确地聚合这些线索到一个全球性的视频表示,我们建议挖掘只有少数选定的前景对象之间的相互作用的Transformer,这产生一个稀疏的范例。我们称所提出的框架为事件自适应网络(EAN),因为这两个关键的设计是自适应的输入视频内容。为了利用局部段内的短期运动,我们提出了一种新的和有效的潜在运动编码(LMC)模块,进一步提高了框架的性能。在几个大规模视频数据集上进行了广泛的实验,例如,Something-to-Something V1&V2、Kinetics和Diving 48验证了我们的模型在低FLOP下实现了最先进或具有竞争力的性能。代码可在https://github.com/tianyuan168326/EAN-Pytorch上获得。

英文摘要:

Efficiently modeling spatial-temporal information in videos is crucial for action recognition. To achieve this goal, state-of-the-art methods typically employ the convolution operator and the dense interaction modules such as non-local blocks. However, these methods cannot accurately fit the diverse events in videos. On the one hand, the adopted convolutions are with fixed scales, thus struggling with events of various scales. On the other hand, the dense interaction modeling paradigm only achieves sub-optimal performance as action-irrelevant parts bring additional noises for the final prediction. In this paper, we propose a unified action recognition framework to investigate the dynamic nature of video content by introducing the following designs. First, when extracting local cues, we generate the spatial-temporal kernels of dynamic-scale to adaptively fit the diverse events. Second, to accurately aggregate these cues into a global video representation, we propose to mine the interactions only among a few selected foreground objects by a Transformer, which yields a sparse paradigm. We call the proposed framework as Event Adaptive Network (EAN) because both key designs are adaptive to the input video content. To exploit the short-term motions within local segments, we propose a novel and efficient Latent Motion Code (LMC) module, further improving the performance of the framework. Extensive experiments on several large-scale video datasets, e.g., Something-to-Something V1&V2, Kinetics, and Diving48, verify that our models achieve state-of-the-art or competitive performances at low FLOPs. Codes are available at: https://github.com/tianyuan168326/EAN-Pytorch.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑