One-shot action recognition allows the recognition of human-performed actions with only a single training example. This can influence human-robot-interaction positively by enabling the robot to react to previously unseen behaviour. We formulate the one-shot action recognition problem as a deep metric learning problem and propose a novel image-based skeleton representation that performs well in a metric learning setting. Therefore, we train a model that projects the image representations into an embedding space. In embedding space the similar actions have a low euclidean distance while dissimilar actions have a higher distance. The one-shot action recognition problem becomes a nearest-neighbor search in a set of activity reference samples. We evaluate the performance of our proposed representation against a variety of other skeleton-based image representations. In addition, we present an ablation study that shows the influence of different embedding vector sizes, losses and augmentation. Our approach lifts the state-of-the-art by 3.3% for the one-shot action recognition protocol on the NTU RGB+D 120 dataset under a comparable training setup. With additional augmentation our result improved over 7.7%.
Warsaw University of Technology（华沙理工大学）；NASK - National Research Institute（NASK国家研究院）
Distance Metric Learning (DML) has attracted much attention in image processing in recent years. This paper analyzes its impact on supervised fine-tuning language models for Natural Language Processing (NLP) classification tasks under few-shot learning settings. We investigated several DML loss functions in training RoBERTa language models on known SentEval Transfer Tasks datasets. We also analyzed the possibility of using proxy-based DML losses during model inference. Our systematic experiments have shown that under few-shot learning settings, particularly proxy-based DML losses can positivel…
Common Deep Metric Learning (DML) datasets specify only one notion of similarity, e.g., two images in the Cars196 dataset are deemed similar if they show the same car model. We argue that depending on the application, users of image retrieval systems have different and changing similarity notions that should be incorporated as easily as possible. Therefore, we present Language-Guided Zero-Shot Deep Metric Learning (LanZ-DML) as a new DML setting in which users control the properties that should be important for image representations without training data by only using natural language. To this…
Recognizing an activity with a single reference sample using metric learning approaches is a promising research field. The majority of few-shot methods focus on object recognition or face-identification. We propose a metric learning approach to reduce the action recognition problem to a nearest neighbor search in embedding space. We encode signals into images and extract features using a deep residual CNN. Using triplet loss, we learn a feature embedding. The resulting encoder transforms features into an embedding space in which closer distances encode similar actions while higher distances en…
Frequency-Semantic Enhanced Variational Autoencoder for Zero-Shot Skeleton-based Action Recognition
University of North Carolina at Charlotte（夏洛特北卡罗来纳大学）；Northern Illinois University（北伊利诺伊大学）；Center for Research in Computer Vision, University of Central Florida（中佛罗里达大学计算机视觉研究中心）
Occlusions are universal disruptions constantly present in the real world. Especially for sparse representations, such as human skeletons, a few occluded points might destroy the geometrical and temporal continuity critically affecting the results. Yet, the research of data-scarce recognition from skeleton sequences, such as one-shot action recognition, does not explicitly consider occlusions despite their everyday pervasiveness. In this work, we explicitly tackle body occlusions for Skeleton-based One-shot Action Recognition (SOAR). We mainly consider two occlusion variants: 1) random occlusi…
This study investigates unsupervised anomaly action recognition, which identifies video-level abnormal-human-behavior events in an unsupervised manner without abnormal samples, and simultaneously addresses three limitations in the conventional skeleton-based approaches: target domain-dependent DNN training, robustness against skeleton errors, and a lack of normal samples. We present a unified, user prompt-guided zero-shot learning framework using a target domain-independent skeleton feature extractor, which is pretrained on a large-scale action recognition dataset. Particularly, during the tra…
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing（测绘遥感信息工程国家重点实验室）；Wuhan University（武汉大学）；Information Systems Technology and Design Pillar（信息系统技术与设计学院）；Singapore University of Technology and Design（新加坡科技与设计大学）；School of Geodesy and Geomatics（测绘学院）；School of Mathematics and Statistics（数学与统计学院）；Wuhan University Shenzhen Research Institute（武汉大学深圳研究院）
Warsaw University of Technology（华沙理工大学）；NASK - National Research Institute（NASK国家研究所）
This paper presents an analysis regarding an influence of the Distance Metric Learning (DML) loss functions on the supervised fine-tuning of the language models for classification tasks. We experimented with known datasets from SentEval Transfer Tasks. Our experiments show that applying the DML loss function can increase performance on downstream classification tasks of RoBERTa-large models in few-shot scenarios. Models fine-tuned with the use of SoftTriple loss can achieve better results than models with a standard categorical cross-entropy loss function by about 2.89 percentage points from 0…
Skeleton-based action recognition receives increasing attention because the skeleton representations reduce the amount of training data by eliminating visual information irrelevant to actions. To further improve the sample efficiency, meta-learning-based one-shot learning solutions were developed for skeleton-based action recognition. These methods find the nearest neighbor according to the similarity between instance-level global average embedding. However, such measurement holds unstable representativity due to inadequate generalized learning on local invariant and noisy features, while intu…
Center for Visual Information Technology（视觉信息技术中心）；IIIT Hyderabad（海得拉巴国际信息技术学院）
We introduce SynSE, a novel syntactically guided generative approach for Zero-Shot Learning (ZSL). Our end-to-end approach learns progressively refined generative embedding spaces constrained within and across the involved modalities (visual, language). The inter-modal constraints are defined between action sequence embedding and embeddings of Parts of Speech (PoS) tagged words in the corresponding action description. We deploy SynSE for the task of skeleton-based action sequence recognition. Our design choices enable SynSE to generalize compositionally, i.e., recognize sequences whose action …
How does one represent an action? How does one describe an action that we have never seen before? Such questions are addressed by the Zero Shot Learning paradigm, where a model is trained on only a subset of classes and is evaluated on its ability to correctly classify an example from a class it has never seen before. In this work, we present a body pose based zero shot action recognition network and demonstrate its performance on the NTU RGB-D dataset. Our model learns to jointly encapsulate visual similarities based on pose features of the action performer as well as similarities in the natu…
Zero-Shot Skeleton-Based Action Recognition With Prototype-Guided Feature Alignment
School of Software Engineering, South China University of Technology（南方科技大学软件工程学院）；South China University of Technology（南方科技大学）；Pazhou Lab（琶洲实验室）；School of Future Technology, South China University of Technology（未来技术学院）；Peng Cheng Laboratory（鹏城实验室）；Key Laboratory of Big Data and Intelligent Robot (South China University of Technology), Ministry of Education（大数据与智能机器人重点实验室）
Singapore University of Technology and Design（新加坡科技设计大学）；Data61/CSIRO（澳大利亚联邦科学与工业研究组织数据61）；The Australian National University（澳大利亚国立大学）
In this paper, we propose a Few-shot Learning pipeline for 3D skeleton-based action recognition by Joint tEmporal and cAmera viewpoiNt alIgnmEnt (JEANIE). To factor out misalignment between query and support sequences of 3D body joints, we propose an advanced variant of Dynamic Time Warping which jointly models each smooth path between the query and support frames to achieve simultaneously the best alignment in the temporal and simulated camera viewpoint spaces for end-to-end learning under the limited few-shot training data. Sequences are encoded with a temporal block encoder based on Simple …
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University of Tübingen（蒂宾根大学）；Massachusetts Institute of Technology（麻省理工学院）
Deep metric learning (DML) enables learning with less supervision through its emphasis on the similarity structure of representations. There has been much work on improving generalization of DML in settings like zero-shot retrieval, but little is known about its implications for fairness. In this paper, we are the first to evaluate state-of-the-art DML methods trained on imbalanced data, and to show the negative impact these representations have on minority subgroup performance when used for downstream tasks. In this work, we first define fairness in DML through an analysis of three properties…
GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognition
School of Cyber Science and Engineering, Southeast University（东南大学网络空间安全学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Purple Mountain Laboratories（紫金山实验室）
In this paper, we study the problem of one-shot skeleton-based action recognition, which poses unique challenges in learning transferable representation from base classes to novel classes, particularly for fine-grained actions. Existing meta-learning frameworks typically rely on the body-level representations in spatial dimension, which limits the generalisation to capture subtle visual differences in the fine-grained label space. To overcome the above limitation, we propose a part-aware prototypical representation for one-shot skeleton-based action recognition. Our method captures skeleton mo…
Zero-Shot Skeleton-based Action Recognition with Dual Visual-Text Alignment
School of Cyber Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院）；Engineering Research Center of Blockchain Application, Supervision And Management (Southeast University), Ministry of Education, China（区块链应用工程研究中心（东南大学））；School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用关键实验室（东南大学））；School of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China（深圳大学计算机科学与软件工程学院）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University, Shenzhen 518060, China（大数据系统计算技术国家工程实验室（深圳大学））；Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen 518060, China（广东省智能信息处理重点实验室（深圳大学））；Purple Mountain Laboratories, Nanjing 210000, China（紫金山实验室）
SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognition
School of Engineering Science, Simon Fraser University（西蒙菲莎大学工程科学学院）；School of Information Science and Technology, Eastern Institute of Technology（东方理工信息科学与技术学院）
The Hong Kong University of Science and Technology（香港科技大学）；Kuaishou Technology（快手科技）
Human actions involve complex pose variations and their 2D projections can be highly ambiguous. Thus 3D spatio-temporal or 4D (i.e., 3D+T) human skeletons, which are photometric and viewpoint invariant, are an excellent alternative to 2D+T skeletons/pixels to improve action recognition accuracy. This paper proposes a new 4D dataset HAA4D which consists of more than 3,300 RGB videos in 300 human atomic action classes. HAA4D is clean, diverse, class-balanced where each class is viewpoint-balanced with the use of 4D skeletons, in which as few as one 4D skeleton per class is sufficient for trainin…
Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Stanford University（斯坦福大学）；Shanghai AI Laboratory（上海人工智能实验室）
CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Manner
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；School of Biomedical Engineering, Southern Medical University（南方医科大学生物医学工程学院）；Zhujiang Hospital, Southern Medical University（南方医科大学珠江医院）；Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局前沿人工智能研究中心）；Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）
CINVESTAV Unidad Guadalajara（国立理工学院高级研究中心瓜达拉哈拉分部）；Tecnológico de Monterrey（蒙特雷理工学院）
Deep Metric Learning (DML) methods have been proven relevant for visual similarity learning. However, they sometimes lack generalization properties because they are trained often using an inappropriate sample selection strategy or due to the difficulty of the dataset caused by a distributional shift in the data. These represent a significant drawback when attempting to learn the underlying data manifold. Therefore, there is a pressing need to develop better ways of obtaining generalization and representation of the underlying manifold. In this paper, we propose a novel approach to DML that we …
Australian National University（澳大利亚国立大学）；Data61/CSIRO（澳大利亚联邦科学与工业研究组织数据61研究所）
We propose a Few-shot Learning pipeline for 3D skeleton-based action recognition by Joint tEmporal and cAmera viewpoiNt alIgnmEnt (JEANIE). To factor out misalignment between query and support sequences of 3D body joints, we propose an advanced variant of Dynamic Time Warping which jointly models each smooth path between the query and support frames to achieve simultaneously the best alignment in the temporal and simulated camera viewpoint spaces for end-to-end learning under the limited few-shot training data. Sequences are encoded with a temporal block encoder based on Simple Spectral Graph …
T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition
Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Inria Center at Université Côte d'Azur（法国蔚蓝海岸大学Inria中心）；Woven by Toyota；Toyota Motor Europe（丰田汽车欧洲公司）
Hakim Sabzevari University（哈基姆萨布泽瓦里大学）；Ferdowsi University of Mashhad (FUM)（费尔多西马什哈德大学）
Deep Metric Learning (DML) learns a non-linear semantic embedding from input data that brings similar pairs together while keeping dissimilar data away from each other. To this end, many different methods are proposed in the last decade with promising results in various applications. The success of a DML algorithm greatly depends on its loss function. However, no loss function is perfect, and it deals only with some aspects of an optimal similarity embedding. Besides, the generalizability of the DML on unseen categories during the test stage is an important matter that is not considered by exi…
IBM Research AI（IBM研究院人工智能部门）；Tel-Aviv University（特拉维夫大学）；Technion（以色列理工学院）
Distance metric learning (DML) has been successfully applied to object classification, both in the standard regime of rich training data and in the few-shot scenario, where each category is represented by only a few examples. In this work, we propose a new method for DML that simultaneously learns the backbone network parameters, the embedding space, and the multi-modal distribution of each of the training categories in that space, in a single end-to-end training process. Our approach outperforms state-of-the-art methods for DML-based object classification on a variety of standard fine-grained…
Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition
State Key Laboratory of General Artificial Intelligence, Peking University（通用人工智能国家重点实验室，北京大学）；Center for Research in Computer Vision, University of Central Florida（中佛罗里达大学计算机视觉研究中心）
Human affect recognition is a well-established research area with numerous applications, e.g., in psychological care, but existing methods assume that all emotions-of-interest are given a priori as annotated training examples. However, the rising granularity and refinements of the human emotional spectrum through novel psychological theories and the increased consideration of emotions in context brings considerable pressure to data collection and labeling work. In this paper, we conceptualize one-shot recognition of emotions in context -- a new problem aimed at recognizing human affect states …
A metric learning approach for endoscopic kidney stone identification
Centro de Investigación y de Estudios Avanzados（高级研究中心）；Tecnologico de Monterrey（蒙特雷科技大学）；Centre de Recherche en Automatique de Nancy（南锡自动化研究中心）；CNRS（法国国家科学研究中心）；Université de Lorraine（洛林大学）；CHU Nancy（南锡大学附属医院）
How do Foundation Models Compare to Skeleton-Based Approaches for Gesture Recognition in Human-Robot Interaction?
Chair for Computer Vision, RWTH Aachen University（计算机视觉教研室，亚琛RWTH大学）；Robert Bosch GmbH, Corporate Research & Bosch Center for AI（博世公司，博世企业研究院及博世人工智能中心）
Meet JEANIE: a Similarity Measure for 3D Skeleton Sequences via Temporal-Viewpoint Alignment
the Australian National University (ANU)（澳大利亚国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Curtin University（科廷大学）；Australian National University (ANU)（澳大利亚国立大学）
LMU Munich（慕尼黑大学）；Heidelberg University（海德堡大学）；University of Toronto（多伦多大学）；University of Washington（华盛顿大学）；MIT（麻省理工学院）
Deep Metric Learning (DML) aims to find representations suitable for zero-shot transfer to a priori unknown test distributions. However, common evaluation protocols only test a single, fixed data split in which train and test classes are assigned randomly. More realistic evaluations should consider a broad spectrum of distribution shifts with potentially varying degree and difficulty. In this work, we systematically construct train-test splits of increasing difficulty and present the ooDML benchmark to characterize generalization under out-of-distribution shifts in DML. ooDML is designed to pr…
Temporal action segmentation (TAS) aims to classify and locate actions in the long untrimmed action sequence. With the success of deep learning, many deep models for action segmentation have emerged. However, few-shot TAS is still a challenging problem. This study proposes an efficient framework for the few-shot skeleton-based TAS, including a data augmentation method and an improved model. The data augmentation approach based on motion interpolation is presented here to solve the problem of insufficient data, and can increase the number of samples significantly by synthesizing action sequence…
The Hong Kong University of Science and Technology（香港科技大学）
Classical close-set semantic segmentation networks have limited ability to detect out-of-distribution (OOD) objects, which is important for safety-critical applications such as autonomous driving. Incrementally learning these OOD objects with few annotations is an ideal way to enlarge the knowledge base of the deep learning models. In this paper, we propose an open world semantic segmentation system that includes two modules: (1) an open-set semantic segmentation module to detect both in-distribution and OOD objects. (2) an incremental few-shot learning module to gradually incorporate those OO…
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Southeast University - Monash University Joint Graduate School（东南大学-莫纳什大学联合研究生院）；School of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）
Few-Shot Learning of a Graph-Based Neural Network Model Without Backpropagation
National Technical University "Kharkiv Polytechnic Institute"（克里木·克里姆斯基 Kharkiv 工程学院）；School of Computer and Cyber Sciences（计算机与网络安全学院）；Augusta University（奥古斯塔大学）
