用于时空表征学习的移位块Transformer
Shifted Chunk Transformer for Spatio-Temporal Representational Learning
摘要：
时空表示学习在动作识别、视频对象分割、动作预测等领域得到了广泛的应用。以前的时空表示学习方法主要采用ConvNet或顺序模型，例如，LSTM，学习帧内和帧间特征。最近，Transformer模型已经成功地主导了自然语言处理（NLP），图像分类等的研究。然而，纯粹的Transformer为基础的时空学习可以是昂贵的内存和计算，从一个微小的补丁提取细粒度的功能。为了解决训练困难和增强时空学习，我们构建了一个转移组块Transformer与纯自我注意块。利用NLP中最近的高效Transformer设计，这个移位块Transformer可以从局部小块到全局视频剪辑学习分层时空特征。我们转移的自我注意力也可以有效地模拟复杂的帧间方差。此外，我们建立了一个剪辑编码器的基础上Transformer模型的长期时间依赖性。我们进行了彻底的消融研究，以验证我们的移位块Transformer中的每个组件和超参数，它在Kinetics-400，Kinetics-600，UCF 101和HMDB 51上的表现优于以前的最先进方法。
英文摘要：
Spatio-temporal representational learning has been widely adopted in various fields such as action recognition, video object segmentation, and action anticipation. Previous spatio-temporal representational learning approaches primarily employ ConvNets or sequential models,e.g., LSTM, to learn the intra-frame and inter-frame features. Recently, Transformer models have successfully dominated the study of natural language processing (NLP), image classification, etc. However, the pure-Transformer based spatio-temporal learning can be prohibitively costly on memory and computation to extract fine-grained features from a tiny patch. To tackle the training difficulty and enhance the spatio-temporal learning, we construct a shifted chunk Transformer with pure self-attention blocks. Leveraging the recent efficient Transformer design in NLP, this shifted chunk Transformer can learn hierarchical spatio-temporal features from a local tiny patch to a global video clip. Our shifted self-attention can also effectively model complicated inter-frame variances. Furthermore, we build a clip encoder based on Transformer to model long-term temporal dependencies. We conduct thorough ablation studies to validate each component and hyper-parameters in our shifted chunk Transformer, and it outperforms previous state-of-the-art approaches on Kinetics-400, Kinetics-600, UCF101, and HMDB51.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
