arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.05095cs.CV
收藏

时空注意力是理解视频理解所需的全部吗?

Is Space-Time Attention All You Need for Video Understanding?

Facebook AI(脸书人工智能);Dartmouth College(达特茅斯学院)

Gedas Bertasius, Heng Wang, Lorenzo Torresani

摘要:

我们提出了一种无卷积的视频分类方法,该方法完全基于空间和时间上的自我注意力。我们的方法,名为“TimeTransformer”,适应标准的Transformer架构的视频,使时空特征学习直接从帧级补丁的序列。我们的实验研究比较了不同的自我注意力方案,并表明,“分散注意力”,其中时间注意力和空间注意力分别适用于每个块,导致最好的视频分类精度之间的设计选择考虑。尽管采用了全新的设计,但TimeSmerer在几个动作识别基准测试中取得了最先进的结果,包括在Kinetics-400和Kinetics-600上报告的最佳准确性。最后,与3D卷积网络相比,我们的模型训练速度更快,可以实现更高的测试效率(精度略有下降),并且还可以应用于更长的视频剪辑(超过一分钟)。代码和模型可在https://github.com/facebookresearch/TimeSformer上获得。

英文摘要:

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/facebookresearch/TimeSformer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑