TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting
ShanghaiTech University(上海科技大学);National University of Singapore(新加坡国立大学);Shanghai Engineering Research Center of Intelligent Vision and Imaging(上海智能视觉与成像工程技术研究中心);Shanghai Engineering Research Center of Energy Efficient and Custom AI IC(上海能效与定制人工智能集成电路工程技术研究中心)
Counting repetitive actions are widely seen in human activities such as physical exercise. Existing methods focus on performing repetitive action counting in short videos, which is tough for dealing with longer videos in more realistic scenarios. In the data-driven era, the degradation of such generalization capability is mainly attributed to the lack of long video datasets. To complement this margin, we introduce a new large-scale repetitive action counting dataset covering a wide variety of video lengths, along with more realistic situations where action interruption or action inconsistencies occur in the video. Besides, we also provide a fine-grained annotation of the action cycles instead of just counting annotation along with a numerical value. Such a dataset contains 1,451 videos with about 20,000 annotations, which is more challenging. For repetitive action counting towards more realistic scenarios, we further propose encoding multi-scale temporal correlation with transformers that can take into account both performance and efficiency. Furthermore, with the help of fine-grained annotation of action cycles, we propose a density map regression-based method to predict the action period, which yields better performance with sufficient interpretability. Our proposed method outperforms state-of-the-art methods on all datasets and also achieves better performance on the unseen dataset without fine-tuning. The dataset and code are available.
Repetitive Action Counting with Hybrid Temporal Relation Modeling
School of Computer Science and Information Engineering, Hefei University of Technology (HFUT)(合肥工业大学计算机与信息工程学院);Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education(合肥工业大学大数据知识工程重点实验室(教育部));Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Anhui Zhonghuitong Technology Co., Ltd.(安徽中汇通科技有限公司);School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学技术学院)
TReViS: Temporal Repetition Structure Aware Video Synthesis for Self-supervised Repetitive Action Counting
AIGO Unit, Istituto Italiano di Tecnologia(意大利技术研究院AIGO单元);Department of Computer Science, University of Verona(维罗纳大学计算机科学系);University of Genoa(热那亚大学)
Action Dubber: Timing Audible Actions via Inflectional Flow
South China University of Technology(华南理工大学);City University of Hong Kong(香港城市大学);University of Hong Kong(香港大学);Singapore Management University(新加坡管理大学)