arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.01018cs.CV
收藏

TransRAC:基于Transformers的多尺度时间相关性编码

TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting

ShanghaiTech University(上海科技大学);National University of Singapore(新加坡国立大学);Shanghai Engineering Research Center of Intelligent Vision and Imaging(上海智能视觉与成像工程技术研究中心);Shanghai Engineering Research Center of Energy Efficient and Custom AI IC(上海能效与定制人工智能集成电路工程技术研究中心)

Huazhang Hu, Sixun Dong, Yiqun Zhao, Dongze Lian, Zhengxin Li, Shenghua Gao

摘要:

计算重复动作在人类活动中广泛存在,如体育锻炼。现有的方法集中于执行短视频中的重复动作计数,这对于在更真实的场景中处理较长的视频是困难的。在数据驱动的时代,这种泛化能力的下降主要归因于缺乏长视频数据集。为了补充这一优势,我们引入了一个新的大规模重复动作计数数据集,涵盖了各种视频长度,以及视频中出现动作中断或动作不一致的更现实的情况。此外,我们还提供了动作周期的细粒度注释,而不仅仅是计数注释和数值。这样的数据集包含1,451个视频,约有20,000个注释,这更具挑战性。对于面向更现实场景的重复动作计数,我们进一步提出了使用可以兼顾性能和效率的Transformers对多尺度时间相关性进行编码。此外,在细粒度注释的帮助下,我们提出了一种基于密度图回归的方法来预测动作周期,从而产生更好的性能与足够的可解释性。我们提出的方法在所有数据集上都优于最先进的方法,并且在没有微调的情况下在看不见的数据集上也实现了更好的性能。数据集和代码可用。

英文摘要:

Counting repetitive actions are widely seen in human activities such as physical exercise. Existing methods focus on performing repetitive action counting in short videos, which is tough for dealing with longer videos in more realistic scenarios. In the data-driven era, the degradation of such generalization capability is mainly attributed to the lack of long video datasets. To complement this margin, we introduce a new large-scale repetitive action counting dataset covering a wide variety of video lengths, along with more realistic situations where action interruption or action inconsistencies occur in the video. Besides, we also provide a fine-grained annotation of the action cycles instead of just counting annotation along with a numerical value. Such a dataset contains 1,451 videos with about 20,000 annotations, which is more challenging. For repetitive action counting towards more realistic scenarios, we further propose encoding multi-scale temporal correlation with transformers that can take into account both performance and efficiency. Furthermore, with the help of fine-grained annotation of action cycles, we propose a density map regression-based method to predict the action period, which yields better performance with sufficient interpretability. Our proposed method outperforms state-of-the-art methods on all datasets and also achieves better performance on the unseen dataset without fine-tuning. The dataset and code are available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑