Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition
Key Laboratory of Machine Perception, Shenzhen Graduate School, Peking University（北京大学深圳研究生院机器感知重点实验室）；The School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能系统工程学院）
Tianyu Guo, Hong Liu, Zhan Chen, Mengyuan Liu, Tao Wang, Runwei Ding
In recent years, self-supervised representation learning for skeleton-based action recognition has been developed with the advance of contrastive learning methods. The existing contrastive learning methods use normal augmentations to construct similar positive samples, which limits the ability to explore novel movement patterns. In this paper, to make better use of the movement patterns introduced by extreme augmentations, a Contrastive Learning framework utilizing Abundant Information Mining for self-supervised action Representation (AimCLR) is proposed. First, the extreme augmentations and the Energy-based Attention-guided Drop Module (EADM) are proposed to obtain diverse positive samples, which bring novel movement patterns to improve the universality of the learned representations. Second, since directly using extreme augmentations may not be able to boost the performance due to the drastic changes in original identity, the Dual Distributional Divergence Minimization Loss (D$^3$M Loss) is proposed to minimize the distribution divergence in a more gentle way. Third, the Nearest Neighbors Mining (NNM) is proposed to further expand positive samples to make the abundant information mining process more reasonable. Exhaustive experiments on NTU RGB+D 60, PKU-MMD, NTU RGB+D 120 datasets have verified that our AimCLR can significantly perform favorably against state-of-the-art methods under a variety of evaluation protocols with observed higher quality action representations. Our code is available at https://github.com/Levigty/AimCLR.
Self-supervised skeleton-based action recognition with contrastive learning has attracted much attention. Recent literature shows that data augmentation and large sets of contrastive pairs are crucial in learning such representations. In this paper, we found that directly extending contrastive pairs based on normal augmentations brings limited returns in terms of performance, because the contribution of contrastive pairs from the normal data augmentation to the loss get smaller as training progresses. Therefore, we delve into hard contrastive pairs for contrastive learning. Motivated by the su…
This paper strives for self-supervised learning of a feature space suitable for skeleton-based action recognition. Our proposal is built upon learning invariances to input skeleton representations and various skeleton augmentations via a noise contrastive estimation. In particular, we propose inter-skeleton contrastive learning, which learns from multiple different input skeleton representations in a cross-contrastive manner. In addition, we contribute several skeleton-specific spatial and temporal augmentations which further encourage the model to learn the spatio-temporal dynamics of skeleto…
Tianjin University（天津大学）；Key Research Center for Surface Monitoring and Analysis of Cultural Relics (SMARC), State Administration of Cultural Heritage, China（中国国家文物局文物表面监测与分析重点研究中心）；University of South Carolina（南卡罗来纳大学）
Skeleton-based human action recognition has been drawing more interest recently due to its low sensitivity to appearance changes and the accessibility of more skeleton data. However, even the 3D skeletons captured in practice are still sensitive to the viewpoint and direction gave the occlusion of different human-body joints and the errors in human joint localization. Such view variance of skeleton data may significantly affect the performance of action recognition. To address this issue, we propose in this paper a new view-invariant representation learning approach, without any manual action …
Nanjing University of Science and Technology（南京理工大学）
Contrastive learning has been successfully leveraged to learn action representations for addressing the problem of semi-supervised skeleton-based action recognition. However, most contrastive learning-based methods only contrast global features mixing spatiotemporal information, which confuses the spatial- and temporal-specific information reflecting different semantic at the frame level and joint level. Thus, we propose a novel Spatiotemporal Decouple-and-Squeeze Contrastive Learning (SDS-CL) framework to comprehensively learn more abundant representations of skeleton-based actions by jointly…
University of Technology Sydney（悉尼科技大学）；University of Queensland（昆士兰大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）
In this work, we propose a new Dual Min-Max Games (DMMG) based self-supervised skeleton action recognition method by augmenting unlabeled data in a contrastive learning framework. Our DMMG consists of a viewpoint variation min-max game and an edge perturbation min-max game. These two min-max games adopt an adversarial paradigm to perform data augmentation on the skeleton sequences and graph-structured body joints, respectively. Our viewpoint variation min-max game focuses on constructing various hard contrastive pairs by generating skeleton sequences from various viewpoints. These hard contras…
Action recognition via 3D skeleton data is an emerging important topic in these years. Most existing methods either extract hand-crafted descriptors or learn action representations by supervised learning paradigms that require massive labeled data. In this paper, we for the first time propose a contrastive action learning paradigm named AS-CAL that can leverage different augmentations of unlabeled skeleton data to learn action representations in an unsupervised manner. Specifically, we first propose to contrast similarity between augmented instances (query and key) of the input skeleton sequen…
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
The self-supervised pretraining paradigm has achieved great success in skeleton-based action recognition. However, these methods treat the motion and static parts equally, and lack an adaptive design for different parts, which has a negative impact on the accuracy of action recognition. To realize the adaptive action modeling of both parts, we propose an Actionlet-Dependent Contrastive Learning method (ActCLR). The actionlet, defined as the discriminative subset of the human skeleton, effectively decomposes motion regions for better action modeling. In detail, by contrasting with the static an…
Zhejiang University（浙江大学）；Harvard Medical School（哈佛医学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Skeleton-based action recognition is widely used in varied areas, e.g., surveillance and human-machine interaction. Existing models are mainly learned in a supervised manner, thus heavily depending on large-scale labeled data which could be infeasible when labels are prohibitively expensive. In this paper, we propose a novel Contrast-Reconstruction Representation Learning network (CRRL) that simultaneously captures postures and motion dynamics for unsupervised skeleton-based action recognition. It mainly consists of three parts: Sequence Reconstructor, Contrastive Motion Learner, and Informati…
DoGCLR: Dominance-Game Contrastive Learning Network for Skeleton-Based Action Recognition
Shenzhen University（深圳大学）；the Institute of Intelligence Information Processing（智能信息处理研究院）；the Guangdong Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）
Graph convolutional networks (GCNs) have been very successful in modeling non-Euclidean data structures, like sequences of body skeletons forming actions modeled as spatio-temporal graphs. Most GCN-based action recognition methods use deep feed-forward networks with high computational complexity to process all skeletons in an action. This leads to a high number of floating point operations (ranging from 16G to 100G FLOPs) to process a single sample, making their adoption in restricted computation application scenarios infeasible. In this paper, we propose a temporal attention module (TAM) for …
Part Aware Contrastive Learning for Self-Supervised Action Recognition
Wuhan University of Science and Technology（武汉科技大学）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；University of Central Florida（中佛罗里达大学）；Peking University（北京大学）
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Artificial Intelligence Industrial Technology Research Institute, Nanjing Institute of Technology（南京工程学院人工智能产业技术研究院）；School of Computer Science, Nanjing University（南京大学计算机科学学院）
针对无监督骨骼动作识别中随机增强难以产生直接语义扰动且会软化正负对的问题，提出 A²MC，利用攻击增强生成困难正样本并由 PNM 混合生成困难负样本，在三个公开数据集上达到与 SOTA 相当的性能。
Shandong Normal University（山东师范大学）；Deakin University（迪肯大学）
For pursuing accurate skeleton-based action recognition, most prior methods use the strategy of combining Graph Convolution Networks (GCNs) with attention-based methods in a serial way. However, they regard the human skeleton as a complete graph, resulting in less variations between different actions (e.g., the connection between the elbow and head in action ``clapping hands''). For this, we propose a novel Contrastive GCN-Transformer Network (ConGT) which fuses the spatial and temporal modules in a parallel way. The ConGT involves two parallel streams: Spatial-Temporal Graph Convolution strea…
Wavelet-Decoupling Contrastive Enhancement Network for Fine-Grained Skeleton-Based Action Recognition
Beijing Engineering Research Center of Mixed Reality and Advanced Display（北京混合现实与先进显示工程研究中心）；School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院）
Nanjing University of Science and Technology（南京理工大学）
Most semi-supervised skeleton-based action recognition approaches aim to learn the skeleton action representations only at the joint level, but neglect the crucial motion characteristics at the coarser-grained body (e.g., limb, trunk) level that provide rich additional semantic information, though the number of labeled data is limited. In this work, we propose a novel Pyramid Self-attention Polymerization Learning (dubbed as PSP Learning) framework to jointly learn body-level, part-level, and joint-level action representations of joint and motion data containing abundant and complementary sema…
In this work, we propose a Cross-view Contrastive Learning framework for unsupervised 3D skeleton-based action Representation (CrosSCLR), by leveraging multi-view complementary supervision signal. CrosSCLR consists of both single-view contrastive learning (SkeletonCLR) and cross-view consistent knowledge mining (CVC-KM) modules, integrated in a collaborative learning manner. It is noted that CVC-KM works in such a way that high-confidence positive/negative samples and their distributions are exchanged among views according to their embedding similarity, ensuring cross-view consistency in terms…
Despite the success of fully-supervised human skeleton sequence modeling, utilizing self-supervised pre-training for skeleton sequence representation learning has been an active field because acquiring task-specific skeleton annotations at large scales is difficult. Recent studies focus on learning video-level temporal and discriminative information using contrastive learning, but overlook the hierarchical spatial-temporal nature of human skeletons. Different from such superficial supervision at the video level, we propose a self-supervised hierarchical pre-training scheme incorporated into a …
Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabularies
School of Cyber Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院，南京，中国）；School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院，南京，中国）
Beihang University（北京航空航天大学）；Zhongguancun Laboratory（中关村实验室）；Hangzhou Innovation Institute of Beihang University（北京航空航天大学杭州创新研究院）
Human action recognition aims at classifying the category of human action from a segment of a video. Recently, people have dived into designing GCN-based models to extract features from skeletons for performing this task, because skeleton representations are much more efficient and robust than other modalities such as RGB frames. However, when employing the skeleton data, some important clues like related items are also discarded. It results in some ambiguous actions that are hard to be distinguished and tend to be misclassified. To alleviate this problem, we propose an auxiliary feature refin…
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机技术研究所）
In this paper, we address self-supervised representation learning from human skeletons for action recognition. Previous methods, which usually learn feature presentations from a single reconstruction task, may come across the overfitting problem, and the features are not generalizable for action recognition. Instead, we propose to integrate multiple tasks to learn more general representations in a self-supervised manner. To realize this goal, we integrate motion prediction, jigsaw puzzle recognition, and contrastive learning to learn skeleton features from different aspects. Skeleton dynamics …
College of Computer Science and Technology, Zhejiang Gongshang University（浙江工商大学计算机科学与技术学院）；Zhejiang Key Lab of E-Commerce（浙江省电子商务重点实验室）
This paper targets unsupervised skeleton-based action representation learning and proposes a new Hierarchical Contrast (HiCo) framework. Different from the existing contrastive-based solutions that typically represent an input skeleton sequence into instance-level features and perform contrast holistically, our proposed HiCo represents the input into multiple-level features and performs contrast in a hierarchical manner. Specifically, given a human skeleton sequence, we represent it into multiple feature vectors of different granularities from both temporal and spatial domains via sequence-to-…
Fraunhofer IPA（弗劳恩霍夫制造工程与自动化研究所）；University of Technology Chemnitz（开姆尼茨工业大学）
Understanding human actions is a crucial problem for service robots. However, the general trend in Action Recognition is developing and testing these systems on structured datasets. That's why this work presents a practical Skeleton-based Action Recognition framework which can be used in realistic scenarios. Our results show that although non-augmented and non-normalized data may yield comparable results on the test split of the dataset, it is far from being useful on another dataset which is a manually collected data.
Zero-Shot Skeleton-Based Action Recognition With Prototype-Guided Feature Alignment
School of Software Engineering, South China University of Technology（南方科技大学软件工程学院）；South China University of Technology（南方科技大学）；Pazhou Lab（琶洲实验室）；School of Future Technology, South China University of Technology（未来技术学院）；Peng Cheng Laboratory（鹏城实验室）；Key Laboratory of Big Data and Intelligent Robot (South China University of Technology), Ministry of Education（大数据与智能机器人重点实验室）
Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition
State Key Laboratory of General Artificial Intelligence, Peking University（通用人工智能国家重点实验室，北京大学）；Center for Research in Computer Vision, University of Central Florida（中佛罗里达大学计算机视觉研究中心）
T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition
Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Inria Center at Université Côte d'Azur（法国蔚蓝海岸大学Inria中心）；Woven by Toyota；Toyota Motor Europe（丰田汽车欧洲公司）
Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）
USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation
Southeast University（东南大学）；Northwestern Polytechnical University（西北工业大学）；Tsinghua University（清华大学）；Nanjing University of Science and Technology（南京理工大学）
TranSG: Transformer-Based Skeleton Graph Prototype Contrastive Learning with Structure-Trajectory Prompted Reconstruction for Person Re-Identification
LILY Research Center, Nanyang Technological University（南洋理工大学LILY研究中心）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
Dynamic hand gesture recognition has attracted increasing interests because of its importance for human computer interaction. In this paper, we propose a new motion feature augmented recurrent neural network for skeleton-based dynamic hand gesture recognition. Finger motion features are extracted to describe finger movements and global motion features are utilized to represent the global movement of hand skeleton. These motion features are then fed into a bidirectional recurrent neural network (RNN) along with the skeleton sequence, which can augment the motion features for RNN and improve the…
Affinity Contrastive Learning for Skeleton-based Human Activity Understanding
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Electronic, Electrical and Communication Engineering, Univeristy of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Moonshot AI
Nanyang Technological University（南洋理工大学）；Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY)（南洋理工大学-不列颠哥伦比亚大学活跃老年生活卓越联合研究中心（LILY））
Recent advances in skeleton-based person re-identification (re-ID) obtain impressive performance via either hand-crafted skeleton descriptors or skeleton representation learning with deep learning paradigms. However, they typically require skeletal pre-modeling and label information for training, which leads to limited applicability of these methods. In this paper, we focus on unsupervised skeleton-based person re-ID, and present a generic Simple Masked Contrastive learning (SimMC) framework to learn effective representations from unlabeled 3D skeletons for person re-ID. Specifically, to fully…
Human skeleton point clouds are commonly used to automatically classify and predict the behaviour of others. In this paper, we use a contrastive self-supervised learning method, SimCLR, to learn representations that capture the semantics of skeleton point clouds. This work focuses on systematically evaluating the effects that different algorithmic decisions (including augmentations, dataset partitioning and backbone architecture) have on the learned skeleton representations. To pre-train the representations, we normalise six existing datasets to obtain more than 40 million skeleton frames. We …
Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly（南洋理工大学-不列颠哥伦比亚大学活跃老年生活卓越联合研究中心）；Nanyang Technological University（南洋理工大学）
Person re-identification (re-ID) via 3D skeletons is an important emerging topic with many merits. Existing solutions rarely explore valuable body-component relations in skeletal structure or motion, and they typically lack the ability to learn general representations with unlabeled skeleton data for person re-ID. This paper proposes a generic unsupervised Skeleton Prototype Contrastive learning paradigm with Multi-level Graph Relation learning (SPC-MGR) to learn effective representations from unlabeled skeletons to perform person re-ID. Specifically, we first construct unified multi-level ske…
