Repetitive Action Counting with Hybrid Temporal Relation Modeling
School of Computer Science and Information Engineering, Hefei University of Technology (HFUT)(合肥工业大学计算机与信息工程学院);Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education(合肥工业大学大数据知识工程重点实验室(教育部));Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Anhui Zhonghuitong Technology Co., Ltd.(安徽中汇通科技有限公司);School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学技术学院)
Kun Li, Xinge Peng, Dan Guo, Xun Yang, Meng Wang
备注:
To be published in IEEE Transactions on Multimedia
Repetitive Action Counting (RAC) aims to count the number of repetitive actions occurring in videos. In the real world, repetitive actions have great diversity and bring numerous challenges (e.g., viewpoint changes, non-uniform periods, and action interruptions). Existing methods based on the temporal self-similarity matrix (TSSM) for RAC are trapped in the bottleneck of insufficient capturing action periods when applied to complicated daily videos. To tackle this issue, we propose a novel method named Hybrid Temporal Relation Modeling Network (HTRM-Net) to build diverse TSSM for RAC. The HTRM-Net mainly consists of three key components: bi-modal temporal self-similarity matrix modeling, random matrix dropping, and local temporal context modeling. Specifically, we construct temporal self-similarity matrices by bi-modal (self-attention and dual-softmax) operations, yielding diverse matrix representations from the combination of row-wise and column-wise correlations. To further enhance matrix representations, we propose incorporating a random matrix dropping module to guide channel-wise learning of the matrix explicitly. After that, we inject the local temporal context of video frames and the learned matrix into temporal correlation modeling, which can make the model robust enough to cope with error-prone situations, such as action interruption. Finally, a multi-scale matrix fusion module is designed to aggregate temporal correlations adaptively in multi-scale matrices. Extensive experiments across intra- and cross-datasets demonstrate that the proposed method not only outperforms current state-of-the-art methods but also exhibits robust capabilities in accurately counting repetitive actions in unseen action categories. Notably, our method surpasses the classical TransRAC method by 20.04\% in MAE and 22.76\% in OBO.
TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting
ShanghaiTech University(上海科技大学);National University of Singapore(新加坡国立大学);Shanghai Engineering Research Center of Intelligent Vision and Imaging(上海智能视觉与成像工程技术研究中心);Shanghai Engineering Research Center of Energy Efficient and Custom AI IC(上海能效与定制人工智能集成电路工程技术研究中心)
Developing a Hybrid Convolutional Neural Network for Automatic Aphid Counting in Sugar Beet Fields
Tobacco Research Institute, Chinese Academy of Agricultural Sciences(中国农业科学院烟草研究所);University of Lincoln(林肯大学);British Beet Research Organisation(英国甜菜研究组织);Lincoln Agri-Robotics Centre, Lincoln Institute for Agri-Food Technology, University of Lincoln(林肯大学林肯农业食品技术研究所林肯农业机器人中心);Lincoln Centre for Autonomous Systems (L-CAS), University of Lincoln(林肯大学林肯自主系统中心)
TReViS: Temporal Repetition Structure Aware Video Synthesis for Self-supervised Repetitive Action Counting
AIGO Unit, Istituto Italiano di Tecnologia(意大利技术研究院AIGO单元);Department of Computer Science, University of Verona(维罗纳大学计算机科学系);University of Genoa(热那亚大学)
University of Electronic Science and Technology of China(电子科技大学);Inception Institute of Artificial Intelligence(人工智能 inception 研究院);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of Pennsylvania(宾夕法尼亚大学);Nanyang Technological University(南洋理工大学)
由于大规模和密度变化,人群计数是一项重要但具有挑战性的任务。最近的研究表明,提取多尺度特征之间的丰富关系并从辅助任务(即,本地化对于这项任务至关重要。然而,如何在统一的网络架构中综合利用这些关系仍然是一个具有挑战性的问题。在本文中,我们提出了一种新的网络结构,称为混合图神经网络(HyGnn),其目标是通过交织的多尺度特征人群密度,以及它的辅助任务(定位)在一起,并在一个图上执行联合推理来缓解这个问题。具体来说,HyGnn集成了一个混合图,将不同尺度的任务特定特征图联合表示为节点,并将两种类型的关系表示为边:(i)用于捕获跨尺度特征依赖关系的多尺度关系和(ii)为计数和定位之间的合作搭建桥梁的互利关系。因此,通过消息传递,HyGnn可以提取节点之间的丰富关系,以获得更强大的表示,从而获得鲁棒和准确的结果。我们的HyGnn在四个具有挑战性的数据集上表现出色:ShanghaiTech Part A,ShanghaiTech Part B,UCF_CC_50和UCF_QNRF,远远优于最先进的方法。