近年来，对比学习作为一种有效的方法，已引起广泛关注，以减少对标注数据的依赖。然而，现有的自监督骨架动作识别方法仍然面临三个主要限制：对视图差异建模不足、缺乏有效的对抗机制以及不可控的增强扰动。为了解决这些问题，我们提出了多视角最小-最大无限骨架-数据游戏对比学习（M3GCLR），一种博弈论对比框架。首先，我们建立了无限骨架-数据游戏（ISG）模型和ISG平衡定理，并进一步提供了严谨的证明，使基于多视角互信息的最小-最大优化成为可能。然后，我们通过多视角旋转增强生成正常-极端数据对，并采用时间平均输入作为中性锚点，以实现结构对齐，从而显式地表征扰动强度。接下来，利用所提出的平衡定理，我们构建了一个强对抗的最小-最大骨架-数据游戏，以鼓励模型挖掘更丰富的动作判别信息。最后，我们引入了双损失平衡优化器来优化游戏平衡，使学习过程在最大化动作相关信息的同时最小化编码冗余，并证明所提出的优化器与ISG模型的等价性。广泛的实验表明，M3GCLR在NTU RGB+D 60（X-Sub，X-View）上实现了三流82.1%、85.8%的准确率，在NTU RGB+D 120（X-Sub，X-Set）上实现了72.3%、75.0%的准确率。在PKU-MMD Part I和II上，分别达到了89.1%、45.2%的三流准确率，所有结果均匹配或优于最先进性能。消融研究证实了每个组件的有效性。
英文摘要：
In recent years, contrastive learning has drawn significant attention as an effective approach to reducing reliance on labeled data. However, existing methods for self-supervised skeleton-based action recognition still face three major limitations: insufficient modeling of view discrepancies, lack of effective adversarial mechanisms, and uncontrollable augmentation perturbations. To tackle these issues, we propose the Multi-view Mini-Max infinite skeleton-data Game Contrastive Learning for skeleton-based action Recognition (M3GCLR), a game-theoretic contrastive framework. First, we establish the Infinite Skeleton-data Game (ISG) model and the ISG equilibrium theorem, and further provide a rigorous proof, enabling mini-max optimization based on multi-view mutual information. Then, we generate normal-extreme data pairs through multi-view rotation augmentation and adopt temporally averaged input as a neutral anchor to achieve structural alignment, thereby explicitly characterizing perturbation strength. Next, leveraging the proposed equilibrium theorem, we construct a strongly adversarial mini-max skeleton-data game to encourage the model to mine richer action-discriminative information. Finally, we introduce the dual-loss equilibrium optimizer to optimize the game equilibrium, allowing the learning process to maximize action-relevant information while minimizing encoding redundancy, and we prove the equivalence between the proposed optimizer and the ISG model. Extensive Experiments show that M3GCLR achieves three-stream 82.1%, 85.8% accuracy on NTU RGB+D 60 (X-Sub, X-View) and 72.3%, 75.0% accuracy on NTU RGB+D 120 (X-Sub, X-Set). On PKU-MMD Part I and II, it attains 89.1%, 45.2% in three-stream respectively, all results matching or outperforming state-of-the-art performance. Ablation studies confirm the effectiveness of each component.
University of Technology Sydney（悉尼科技大学）；University of Queensland（昆士兰大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）
In this work, we propose a new Dual Min-Max Games (DMMG) based self-supervised skeleton action recognition method by augmenting unlabeled data in a contrastive learning framework. Our DMMG consists of a viewpoint variation min-max game and an edge perturbation min-max game. These two min-max games adopt an adversarial paradigm to perform data augmentation on the skeleton sequences and graph-structured body joints, respectively. Our viewpoint variation min-max game focuses on constructing various hard contrastive pairs by generating skeleton sequences from various viewpoints. These hard contras…
SmartMore Corporation（思谋科技）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；South China University of Technology（华南理工大学）
Skeleton-based action recognition has attracted research attentions in recent years. One common drawback in currently popular skeleton-based human action recognition methods is that the sparse skeleton information alone is not sufficient to fully characterize human motion. This limitation makes several existing methods incapable of correctly classifying action categories which exhibit only subtle motion differences. In this paper, we propose a novel framework for employing human pose skeleton and joint-centered light-weight information jointly in a two-stream graph convolutional network, namel…
University of Wollongong（伍伦贡大学）；Hunan University（湖南大学）
Recently, mid-level features have shown promising performance in computer vision. Mid-level features learned by incorporating class-level information are potentially more discriminative than traditional low-level local features. In this paper, an effective method is proposed to extract mid-level features from Kinect skeletons for 3D human action recognition. Firstly, the orientations of limbs connected by two skeleton joints are computed and each orientation is encoded into one of the 27 states indicating the spatial relationship of the joints. Secondly, limbs are combined into parts and the l…
Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Stanford University（斯坦福大学）；Shanghai AI Laboratory（上海人工智能实验室）
Curtin University（科廷大学）；Gwangju Institution of Science and Technology（光州科学技术院）
In skeleton-based action recognition, graph convolutional networks (GCNs), which model human body skeletons using graphical components such as nodes and connections, have achieved remarkable performance recently. However, current state-of-the-art methods for skeleton-based action recognition usually work on the assumption that the completely observed skeletons will be provided. This may be problematic to apply this assumption in real scenarios since there is always a possibility that captured skeletons are incomplete or noisy. In this work, we propose a skeleton-based action recognition method…
Tianjin University（天津大学）；Key Research Center for Surface Monitoring and Analysis of Cultural Relics (SMARC), State Administration of Cultural Heritage, China（中国国家文物局文物表面监测与分析重点研究中心）；University of South Carolina（南卡罗来纳大学）
Skeleton-based human action recognition has been drawing more interest recently due to its low sensitivity to appearance changes and the accessibility of more skeleton data. However, even the 3D skeletons captured in practice are still sensitive to the viewpoint and direction gave the occlusion of different human-body joints and the errors in human joint localization. Such view variance of skeleton data may significantly affect the performance of action recognition. To address this issue, we propose in this paper a new view-invariant representation learning approach, without any manual action …
Key Laboratory of Machine Perception, Shenzhen Graduate School, Peking University（北京大学深圳研究生院机器感知重点实验室）；The School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能系统工程学院）
In recent years, self-supervised representation learning for skeleton-based action recognition has been developed with the advance of contrastive learning methods. The existing contrastive learning methods use normal augmentations to construct similar positive samples, which limits the ability to explore novel movement patterns. In this paper, to make better use of the movement patterns introduced by extreme augmentations, a Contrastive Learning framework utilizing Abundant Information Mining for self-supervised action Representation (AimCLR) is proposed. First, the extreme augmentations and t…
Human skeleton joints are popular for action analysis since they can be easily extracted from videos to discard background noises. However, current skeleton representations do not fully benefit from machine learning with CNNs. We propose "Skepxels" a spatio-temporal representation for skeleton sequences to fully exploit the "local" correlations between joints using the 2D convolution kernels of CNN. We transform skeleton videos into images of flexible dimensions using Skepxels and develop a CNN-based framework for effective human action recognition using the resulting images. Skepxels encode r…
Sun Yat-sen University（中山大学）；State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Peking University, Shenzhen Graduate School（北京大学深圳研究生院）；Sony R&D Center China（索尼中国研发中心）
Navigating Open Set Scenarios for Skeleton-based Action Recognition
School of Robotics, Hunan University（湖南大学机器人学院）；Mercedes-Benz Tech Innovation（梅赛德斯-奔驰技术创新公司）；Institute for Artificial Intelligence, University of Stuttgart（斯图加特大学人工智能研究所）
Shandong Normal University（山东师范大学）；Deakin University（迪肯大学）
For pursuing accurate skeleton-based action recognition, most prior methods use the strategy of combining Graph Convolution Networks (GCNs) with attention-based methods in a serial way. However, they regard the human skeleton as a complete graph, resulting in less variations between different actions (e.g., the connection between the elbow and head in action ``clapping hands''). For this, we propose a novel Contrastive GCN-Transformer Network (ConGT) which fuses the spatial and temporal modules in a parallel way. The ConGT involves two parallel streams: Spatial-Temporal Graph Convolution strea…
In this paper, we focus on unsupervised representation learning for skeleton-based action recognition. Existing approaches usually learn action representations by sequential prediction but they suffer from the inability to fully learn semantic information. To address this limitation, we propose a novel framework named Prototypical Contrast and Reverse Prediction (PCRP), which not only creates reverse sequential prediction to learn low-level information (e.g., body posture at every frame) and high-level pattern (e.g., motion order), but also devises action prototypes to implicitly encode semant…
USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation
Southeast University（东南大学）；Northwestern Polytechnical University（西北工业大学）；Tsinghua University（清华大学）；Nanjing University of Science and Technology（南京理工大学）
Wavelet-Decoupling Contrastive Enhancement Network for Fine-Grained Skeleton-Based Action Recognition
Beijing Engineering Research Center of Mixed Reality and Advanced Display（北京混合现实与先进显示工程研究中心）；School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院）
Cross-Block Fine-Grained Semantic Cascade for Skeleton-Based Sports Action Recognition
School of Automation, Southeast University（东南大学自动化学院）；School of Instrument Science and Engineering, Southeast University（东南大学仪器科学与工程学院）；Department of Computer and Information Science, University of Massachusetts Dartmouth（马萨诸塞大学达特茅斯分校计算机与信息科学系）
In this work, we propose a Cross-view Contrastive Learning framework for unsupervised 3D skeleton-based action Representation (CrosSCLR), by leveraging multi-view complementary supervision signal. CrosSCLR consists of both single-view contrastive learning (SkeletonCLR) and cross-view consistent knowledge mining (CVC-KM) modules, integrated in a collaborative learning manner. It is noted that CVC-KM works in such a way that high-confidence positive/negative samples and their distributions are exchanged among views according to their embedding similarity, ensuring cross-view consistency in terms…
Graph-based reasoning over skeleton data has emerged as a promising approach for human action recognition. However, the application of prior graph-based methods, which predominantly employ whole temporal sequences as their input, to the setting of online inference entails considerable computational redundancy. In this paper, we tackle this issue by reformulating the Spatio-Temporal Graph Convolutional Neural Network as a Continual Inference Network, which can perform step-by-step predictions in time without repeat frame processing. To evaluate our method, we create a continual version of ST-GC…
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Southeast University - Monash University Joint Graduate School（东南大学-莫纳什大学联合研究生院）；School of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）
Sapienza University of Rome（罗马第一大学）；Technical University of Munich（慕尼黑工业大学）
Self-paced learning has been beneficial for tasks where some initial knowledge is available, such as weakly supervised learning and domain adaptation, to select and order the training sample sequence, from easy to complex. However its applicability remains unexplored in unsupervised learning, whereby the knowledge of the task matures during training. We propose a novel HYperbolic Self-Paced model (HYSP) for learning skeleton-based action representations. HYSP adopts self-supervision: it uses data augmentations to generate two views of the same sample, and it learns by matching one (named onlin…
School of Computer Science and Technology（计算机科学与技术学院）；University of Science and Technology of China（中国科学技术大学）
Human activity recognition has drawn considerable attention recently in the field of computer vision due to the development of commodity depth cameras, by which the human activity is represented as a sequence of 3D skeleton postures. Assuming human body 3D joint locations of an activity lie on a manifold, the problem of recognizing human activity is formulated as the computation of activity manifold-manifold distance (AMMD). In this paper, we first design an efficient division method to decompose a manifold into ordered continuous maximal linear patches (CMLPs) that denote meaningful action sn…
