Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabularies
School of Cyber Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院，南京，中国）；School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院，南京，中国）
Jidong Kuang, Hongsong Wang, Jie Gui, Yuan Yan Tang, James Tin-Yau Kwok
Skeleton data used for action recognition are acquired from a wide range of sources, including depth sensors, marker-based motion capture systems, and 2D/3D pose estimators. These sources yield skeletons that differ in joint number, skeletal topology, and coordinate dimensionality, making skeleton data inherently heterogeneous. However, previous works overlook the data heterogeneity of skeletons and solely construct models using homogeneous skeletons. Moreover, open-vocabulary action recognition is also essential for real-world applications. To this end, this work studies the challenging problem of heterogeneous skeleton-based action recognition with open vocabularies. We construct a large-scale Heterogeneous Open-Vocabulary (HOV) Skeleton dataset by integrating and refining multiple representative large-scale skeleton-based action datasets. To address universal skeleton-based action recognition, we propose a Transformer-based model that standardizes heterogeneous skeletons into a unified representation, encodes multi-modal skeleton embeddings with a two-stream motion encoder to learn spatio-temporal action representations, and maps them to a semantic space through multi-grained motion-text alignment. The alignment incorporates contrastive learning at three levels: global instance alignment, stream-specific alignment, and fine-grained alignment. Extensive experiments on popular benchmarks with heterogeneous skeleton data demonstrate both the effectiveness and the generalization ability of the proposed method. Code is available at https://github.com/jidongkuang/Universal-Skeleton.
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及交叉应用关键实验室（东南大学））；School of Cyber Science and Engineering, Southeast University, Nanjing 210096, China（东南大学网络科学与工程学院）；Engineering Research Center of Blockchain Application, Supervision And Management (Southeast University), Ministry of Education, China（区块链应用、监督与管理工程研究中心（东南大学））；Purple Mountain Laboratories, Nanjing 210000, China（紫金山实验室）
Skeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models
KTH Royal Institute of Technology（皇家理工学院）；Lancaster University（兰卡斯特大学）；Hebei University of Technology（河北工业大学）；Nanyang Technological University（南洋理工大学）；Shenzhen MSU-BIT University（深圳MSU-BIT大学）；VinUniversity（文大学）
Single-Stream Multi-Feature Fusion with Temporal Robustness for Gait Emotion Recognition
College of Computer and Information Science, Southwest University（西南大学计算机与信息科学学院）；College of Physical Science and Technology, Southwest University（西南大学物理科学与技术学院）；Wisesoft Inc.（四川川大智胜软件股份有限公司）
S3T-Former: A Purely Spike-Driven State-Space Topology Transformer for Skeleton Action Recognition
School of Information and Communication Engineering（信息与通信工程学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；International School（国际学校）；School of Economics and Management（经济管理学院）
Universal Skeleton Understanding via Differentiable Rendering and MLLMs
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China（人工智能通用基础理论国家重点实验室，北京大学深圳研究生院，中国）；Tencent（腾讯）；Nanjing University of Aeronautics（南京航空航天大学）
Nanyang Technological University（南洋理工大学）；Institute for Infocomm Research（资讯通信研究院）；City University of Hong Kong（香港城市大学）
The articulated and complex nature of human actions makes the task of action recognition difficult. One approach to handle this complexity is dividing it to the kinetics of body parts and analyzing the actions based on these partial descriptors. We propose a joint sparse regression based learning method which utilizes the structured sparsity to model each action as a combination of multimodal features from a sparse set of body parts. To represent dynamics and appearance of parts, we employ a heterogeneous set of depth and skeleton based features. The proper structure of multimodal multipart fe…
Universidad de Zaragoza（萨拉戈萨大学）；Bitbrain Technologies（比特脑科技）
Hand action recognition is a special case of action recognition with applications in human-robot interaction, virtual reality or life-logging systems. Building action classifiers able to work for such heterogeneous action domains is very challenging. There are very subtle changes across different actions from a given application but also large variations across domains (e.g. virtual reality vs life-logging). This work introduces a novel skeleton-based hand motion representation model that tackles this problem. The framework we propose is agnostic to the application domain or camera recording v…
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；University of Central Florida（中佛罗里达大学）
Most existing gait recognition methods are appearance-based, which rely on the silhouettes extracted from the video data of human walking activities. The less-investigated skeleton-based gait recognition methods directly learn the gait dynamics from 2D/3D human skeleton sequences, which are theoretically more robust solutions in the presence of appearance changes caused by clothes, hairstyles, and carrying objects. However, the performance of skeleton-based solutions is still largely behind the appearance-based ones. This paper aims to close such performance gap by proposing a novel network mo…
PlantPose: Universal Plant Skeleton Estimation via Tree-constrained Graph Generation
Graduate School of Information Science and Technology, Osaka University（大阪大学信息科学与技术研究生院）；Phytometrics（Phytometrics公司）；Institute of Transformative Bio-Molecules, Nagoya University（名古屋大学变革生物分子研究所）
University of Verona（维罗纳大学）；Università di Modena e Reggio Emilia（摩德纳雷焦艾米利亚大学）；University of Central Florida（中佛罗里达大学）；University of Science, Ho Chi Minh City（胡志明市科学大学）
Gesture recognition is a fundamental tool to enable novel interaction paradigms in a variety of application scenarios like Mixed Reality environments, touchless public kiosks, entertainment systems, and more. Recognition of hand gestures can be nowadays performed directly from the stream of hand skeletons estimated by software provided by low-cost trackers (Ultraleap) and MR headsets (Hololens, Oculus Quest) or by video processing software modules (e.g. Google Mediapipe). Despite the recent advancements in gesture and action recognition from skeletons, it is unclear how well the current state-…
Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment
Tsinghua University（清华大学）；Beihang University（北京航空航天大学）；Children’s Hospital, Capital Institute of Pediatrics（首都儿科研究所附属儿童医院）；Zhongguancun Laboratory（中关村实验室）
