arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.09675cs.CV
收藏

一种基于时间编码和深度压缩的实时的动作表示方法

A Real-time Action Representation with Temporal Encoding and Deep Compression

Beijing University of Posts and Telecommunications(北京邮电大学);South China University of Technology(华南理工大学);MIT-Watson AI Lab(麻省理工学院沃森人工智能实验室)

Kun Liu, Wu Liu, Huadong Ma, Mingkui Tan, Chuang Gan

摘要:

深度神经网络在基于视频的动作识别方面取得了显著的成功。然而,大多数现有的方法不能部署在实践中,由于高的计算成本。为了解决这个问题,我们提出了一种新的实时的【形】卷积结构,称为时间卷积3D网络(T-C3 D),用于动作表示。T-C3 D以分层多粒度方式学习视频动作表示,同时获得高处理速度。具体来说,我们提出了一个残差3D卷积神经网络(CNN)来捕获单个帧的外观和连续帧之间的运动的互补信息。基于此CNN,我们开发了一种新的时间编码方法来探索整个视频的时间动态。此外,我们将深度压缩技术与T-C3 D集成,通过减小模型的大小来进一步加速模型的部署。通过这些方法,可以避免在进行推理时的繁重计算,这使得该方法能够在保持有希望的性能的同时处理超过实时的【形】速度的视频。我们的方法在UCF 101动作识别基准测试中与最先进的实时的【形】方法相比,在准确率方面实现了5.4%的明显改进,在推理速度方面提高了2倍,存储模型小于5 MB。我们验证我们的方法,通过研究其动作表示性能在四个不同的基准在三个不同的任务。大量的实验表明,识别性能的国家的最先进的方法。源代码和预训练模型可在https://github.com/tc3d上公开获得。

英文摘要:

Deep neural networks have achieved remarkable success for video-based action recognition. However, most of existing approaches cannot be deployed in practice due to the high computational cost. To address this challenge, we propose a new real-time convolutional architecture, called Temporal Convolutional 3D Network (T-C3D), for action representation. T-C3D learns video action representations in a hierarchical multi-granularity manner while obtaining a high process speed. Specifically, we propose a residual 3D Convolutional Neural Network (CNN) to capture complementary information on the appearance of a single frame and the motion between consecutive frames. Based on this CNN, we develop a new temporal encoding method to explore the temporal dynamics of the whole video. Furthermore, we integrate deep compression techniques with T-C3D to further accelerate the deployment of models via reducing the size of the model. By these means, heavy calculations can be avoided when doing the inference, which enables the method to deal with videos beyond real-time speed while keeping promising performance. Our method achieves clear improvements on UCF101 action recognition benchmark against state-of-the-art real-time methods by 5.4% in terms of accuracy and 2 times faster in terms of inference speed with a less than 5MB storage model. We validate our approach by studying its action representation performance on four different benchmarks over three different tasks. Extensive experiments demonstrate comparable recognition performance to the state-of-the-art methods. The source code and the pre-trained models are publicly available at https://github.com/tc3d.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑