arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.07233cs.CV
收藏

基于混合时序关系建模的重复动作计数

Repetitive Action Counting with Hybrid Temporal Relation Modeling

School of Computer Science and Information Engineering, Hefei University of Technology (HFUT)(合肥工业大学计算机与信息工程学院);Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education(合肥工业大学大数据知识工程重点实验室(教育部));Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Anhui Zhonghuitong Technology Co., Ltd.(安徽中汇通科技有限公司);School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学技术学院)

Kun Li, Xinge Peng, Dan Guo, Xun Yang, Meng Wang

AI总结:

针对复杂视频中重复动作周期捕获不足的问题,提出HTRM-Net,通过双模态TSSM、随机矩阵丢弃、局部上下文与多尺度融合提升计数鲁棒性,并显著优于TransRAC。

摘要:

重复动作计数(Repetitive Action Counting,RAC)旨在统计视频中发生的重复动作数量。在现实世界中,重复动作具有极大的多样性,并带来诸多挑战(例如视角变化、非均匀周期和动作中断)。现有的用于RAC的基于时序自相似矩阵(temporal self-similarity matrix,TSSM)的方法在应用于复杂日常视频时,陷入了动作周期捕获不足的瓶颈。为了解决这一问题,我们提出了一种名为混合时序关系建模网络(Hybrid Temporal Relation Modeling Network,HTRM-Net)的新方法,以为RAC构建多样化的TSSM。HTRM-Net主要由三个关键组件组成:双模态时序自相似矩阵建模、随机矩阵丢弃和局部时序上下文建模。具体而言,我们通过双模态(自注意力和双softmax)操作构建时序自相似矩阵,从行级与列级相关性的组合中产生多样化的矩阵表示。为进一步增强矩阵表示,我们提出引入随机矩阵丢弃模块,以显式引导矩阵的通道级学习。之后,我们将视频帧的局部时序上下文和学习到的矩阵注入时序相关性建模,这可以使模型足够鲁棒,以应对容易出错的情况,例如动作中断。最后,设计了一个多尺度矩阵融合模块,以在多尺度矩阵中自适应地聚合时序相关性。跨数据集内部和跨数据集的大量实验表明,所提出的方法不仅优于当前最先进的方法,而且在准确计数未见动作类别中的重复动作方面表现出鲁棒能力。值得注意的是,我们的方法在MAE上超越经典TransRAC方法20.04%,在OBO上超越22.76%。

英文摘要:

Repetitive Action Counting (RAC) aims to count the number of repetitive actions occurring in videos. In the real world, repetitive actions have great diversity and bring numerous challenges (e.g., viewpoint changes, non-uniform periods, and action interruptions). Existing methods based on the temporal self-similarity matrix (TSSM) for RAC are trapped in the bottleneck of insufficient capturing action periods when applied to complicated daily videos. To tackle this issue, we propose a novel method named Hybrid Temporal Relation Modeling Network (HTRM-Net) to build diverse TSSM for RAC. The HTRM-Net mainly consists of three key components: bi-modal temporal self-similarity matrix modeling, random matrix dropping, and local temporal context modeling. Specifically, we construct temporal self-similarity matrices by bi-modal (self-attention and dual-softmax) operations, yielding diverse matrix representations from the combination of row-wise and column-wise correlations. To further enhance matrix representations, we propose incorporating a random matrix dropping module to guide channel-wise learning of the matrix explicitly. After that, we inject the local temporal context of video frames and the learned matrix into temporal correlation modeling, which can make the model robust enough to cope with error-prone situations, such as action interruption. Finally, a multi-scale matrix fusion module is designed to aggregate temporal correlations adaptively in multi-scale matrices. Extensive experiments across intra- and cross-datasets demonstrate that the proposed method not only outperforms current state-of-the-art methods but also exhibits robust capabilities in accurately counting repetitive actions in unseen action categories. Notably, our method surpasses the classical TransRAC method by 20.04\% in MAE and 22.76\% in OBO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑