Micro-gesture Online Recognition using Learnable Query Points
School of Computer Science and Information Engineering（计算机与信息工程学院）；School of Artificial Intelligence（人工智能学院）；Hefei University of Technology (HFUT)（合肥工业大学）；Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education（大数据知识工程教育部重点实验室）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Zhejiang University（浙江大学）；Anhui Zhonghuitong Technology Co., Ltd.（安徽中汇通科技有限公司）
Pengyu Liu, Fei Wang, Kun Li, Guoliang Chen, Yanyan Wei, Shengeng Tang, Zhiliang Wu, Dan Guo
备注：
Technical Report of HFUT-VUT for the MiGA challenge at IJCAI 2024
In this paper, we briefly introduce the solution developed by our team, HFUT-VUT, for the Micro-gesture Online Recognition track in the MiGA challenge at IJCAI 2024. The Micro-gesture Online Recognition task involves identifying the category and locating the start and end times of micro-gestures in video clips. Compared to the typical Temporal Action Detection task, the Micro-gesture Online Recognition task focuses more on distinguishing between micro-gestures and pinpointing the start and end times of actions. Our solution ranks 2nd in the Micro-gesture Online Recognition track.
Identity-free Artificial Emotional Intelligence via Micro-Gesture Understanding
Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学（LUT））；Great Bay University（大湾区大学）；Imperial College London（伦敦帝国理工学院）；Technical University of Munich（慕尼黑工业大学）
We introduce a new dataset for the emotional artificial intelligence research: identity-free video dataset for Micro-Gesture Understanding and Emotion analysis (iMiGUE). Different from existing public datasets, iMiGUE focuses on nonverbal body gestures without using any identity information, while the predominant researches of emotion analysis concern sensitive biometric data, like face and speech. Most importantly, iMiGUE focuses on micro-gestures, i.e., unintentional behaviors driven by inner feelings, which are different from ordinary scope of gestures from other gesture datasets which are …
TU Munich（慕尼黑工业大学）；LMU Munich（慕尼黑大学）；University of Tuebingen（图宾根大学）；Intel Labs Europe（英特尔欧洲实验室）；Intel Deutschland GmbH（英特尔德国有限公司）
The use of hand gestures provides a natural alternative to cumbersome interface devices for Human-Computer Interaction (HCI) systems. However, real-time recognition of dynamic micro hand gestures from video streams is challenging for in-vehicle scenarios since (i) the gestures should be performed naturally without distracting the driver, (ii) micro hand gestures occur within very short time intervals at spatially constrained areas, (iii) the performed gesture should be recognized only once, and (iv) the entire architecture should be designed lightweight as it will be deployed to an embedded sy…
Online Micro-gesture Recognition Using Data Augmentation and Spatial-Temporal Attention
School of Computer Science and Information Engineering, School of Artificial Intelligence, Hefei University of Technology (HFUT)（计算机科学与信息工程学院、人工智能学院，合肥工业大学）；ReLER, CCAI, Zhejiang University, China（ReLER、CCAI、浙江大学，中国）；Institute of Advanced Technology, University of Science and Technology of China（先进技术研究院，中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China（人工智能研究院，合肥综合性国家科学中心，中国）；Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education（大数据知识工程重点实验室（HFUT），教育部）；Xinsight Lab, Research Institute, Hefei Zhongjuyuan Intelligent Technology Co., Ltd., China（Xinsight实验室、研究院，合肥中 jouyuan智能科技有限公司，中国）
Prototype Learning for Micro-gesture Classification
Hefei University of Technology (HFUT)（合肥工业大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Zhejiang University（浙江大学）；Anhui Zhonghuitong Technology Co., Ltd.（安徽中汇通科技有限公司）
MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion
University College London (UCL)（伦敦大学学院）；School of Computer Science（计算机科学学院）；Information Engineering, School of Artificial Intelligence, Hefei University of Technology (HFUT)（信息工程学院，人工智能学院，合肥工业大学）；ReLER, CCAI, Zhejiang University, China（ReLER、CCAI、浙江大学，中国）；Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education（大数据知识工程重点实验室（HFUT），教育部）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China（人工智能研究所，合肥综合性国家科学中心，中国）；Xinsight Lab, Research Institute, Hefei Zhongjuyuan Intelligent Technology Co., Ltd., China（Xinsight实验室，研究院，合肥中睿智能科技有限公司，中国）
Hybrid-supervised Hypergraph-enhanced Transformer for Micro-gesture Based Emotion Recognition
School of Electronics and Information at Northwestern Polytechnical University（西北工业大学电子与信息学院）；Innovation Center NPU Chongqing（西北工业大学重庆创新中心）；University of Oulu（奥卢大学）
Spatial-Temporal Decoupled Adapter for Micro-gesture Online Recognition
Hefei University of Technology（合肥工业大学）；United Arab Emirates University（阿联酋大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究院，合肥国家综合科学中心）；Anhui Evolution Technology Co., Ltd.（安徽进化科技有限公司）
Multi-Track Multimodal Learning on iMiGUE: Micro-Gesture and Emotion Recognition
Russian - Armenian University（俄罗斯-亚美尼亚大学）；Sber AI Lab（Sber AI实验室）；HSE University（俄罗斯高等经济大学）；Innopolis University（Innopolis大学）；ISP RAS Research Center for Trusted Artificial Intelligence（俄罗斯科学院可信人工智能研究中心）
University of Electronic Science and Technology of China（电子科技大学）；University of Cincinnati（辛辛那提大学）
Micro-drones can be integrated into various industrial applications but are constrained by their computing power and expert pilots, a secondary challenge. This study presents a computationally-efficient deep convolutional neural network that utilizes Gabor filters and spatial separable convolutions with low computational complexities. An attention module is integrated with the model to complement the performance. Further, perception-based action space and trajectory generators are integrated with the model's predictions for intuitive navigation. The computationally-efficient model aids a human…
WaveFormer: A Lightweight Transformer Model for sEMG-based Gesture Recognition
Integrated Systems Laboratory, ETH Zurich（苏黎世联邦理工学院集成系统实验室）；DEI, University of Bologna（博洛尼亚大学工程学院）；DIEF, University of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学经济工程学院）
本文提出轻量级 Transformer 架构 WaveFormer，利用可学习小波变换和 WaveletConv 融合 sEMG 时频特征，以更少参数实现高准确率与 CPU 端低延迟实时手势识别。
Self-supervised Learning Matters: A Simple Ensemble Solution for Micro-Gesture Recognition
Hefei University of Technology（合肥工业大学）；United Arab Emirates University（阿拉伯联合酋长国大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Anhui Evolution Technology Co., Ltd.（安徽进化科技有限公司）；Nanyang Technological University（南洋理工大学）；University College London（伦敦大学学院）；The University of Sydney（悉尼大学）；Beijing QBoson Quantum Technology Co., Ltd.（北京量子芯科技有限公司）
Active Inference for Micro-Gesture Recognition: EFE-Guided Temporal Sampling and Adaptive Learning
Tianjin Normal University（天津师范大学）；Shenzhen University（深圳大学）；The State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；School of New Media and Communications, Tianjin University（天津大学新媒体与传播学院）
OMG-Bench: A New Challenging Benchmark for Skeleton-based Online Micro Hand Gesture Recognition
School of Systems Science and Engineering, Sun Yat-sen University（中山大学系统科学与工程学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Jiao Tong University（上海交通大学）；Nankai University（南开大学）；Defense Innovation Institute, Academy of Military Sciences（军事科学院国防创新研究院）；Tianjin Artificial Intelligence Innovation Center（天津人工智能创新中心）
A Multi-Modal Framework with Cross-Subject Pseudo-Labeling and Semantic Alignment for Micro-Gesture Recognition
School of Computer Science and Information Engineering, Hefei University of Technology (HFUT)（合肥工业大学计算机科学与信息工程学院）；School of Computer Science, University of Auckland (UOA)（奥克兰大学计算机科学学院）
Hand gestures are a natural means of interaction in Augmented Reality and Virtual Reality (AR/VR) applications. Recently, there has been an increased focus on removing the dependence of accurate hand gesture recognition on complex sensor setup found in expensive proprietary devices such as the Microsoft HoloLens, Daqri and Meta Glasses. Most such solutions either rely on multi-modal sensor data or deep neural networks that can benefit greatly from abundance of labelled data. Datasets are an integral part of any deep learning based research. They have been the principal reason for the substanti…
MSF-Mamba: Motion-aware State Fusion Mamba for Efficient Micro-Gesture Recognition
Computer Vision and Pattern Recognition Laboratory, Department of Computational Engineering, Lappeenranta-Lahti University of Technology LUT（计算机视觉与模式识别实验室，计算工程系，拉佩兰塔-拉赫蒂理工大学）；School of Electrical and Information Engineering, Tianjin University（电气与信息工程学院，天津大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电子与电气工程学院，南洋理工大学）；Faculty of Information Technology, Brno University of Technology（信息技术学院，布拉格技术大学）
Due to the mass advancement in ubiquitous technologies nowadays, new pervasive methods have come into the practice to provide new innovative features and stimulate the research on new human-computer interactions. This paper presents a hand gesture recognition method that utilizes the smartphone's built-in speakers and microphones. The proposed system emits an ultrasonic sonar-based signal (inaudible sound) from the smartphone's stereo speakers, which is then received by the smartphone's microphone and processed via a Convolutional Neural Network (CNN) for Hand Gesture Recognition. Data augment…
GMoT: Gated Motion-Aware Tokenization for Fine-Grained Micro-Gesture Video Reasoning with Multimodal LLMs
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Great Bay University（大湾区大学）；Dongguan Key Laboratory for Intelligence and Information Technology（东莞市智能信息技术重点实验室）
Sign(ed) languages use gestures, such as hand or head movements, for communication. Sign language recognition is an assistive technology for individuals with hearing disability and its goal is to improve such individuals' life quality by facilitating their social involvement. Since sign languages are vastly varied in alphabets, as known as signs, a sign recognition software should be capable of handling eight different types of sign combinations, e.g. numbers, letters, words and sentences. Due to the intrinsic complexity and diversity of symbolic gestures, recognition algorithms need a compreh…
Although psychological research indicates that bodily expressions convey important affective information, to date research in emotion recognition focused mainly on facial expression or voice analysis. In this paper we propose an approach to realtime automatic emotion recognition from body movements. A set of postural, kinematic, and geometrical features are extracted from sequences 3D skeletons and fed to a multi-class SVM classifier. The proposed method has been assessed on data acquired through two different systems: a professionalgrade optical motion capture system, and Microsoft Kinect. Th…
Nitte Meenakshi Institute of Technology（尼特·米纳克希技术学院）
In this article gesture recognition and speech recognition applications are implemented on embedded systems with Tiny Machine Learning (TinyML). It features 3-axis accelerometer, 3-axis gyroscope and 3-axis magnetometer. The gesture recognition,provides an innovative approach nonverbal communication. It has wide applications in human-computer interaction and sign language. Here in the implementation of hand gesture recognition, TinyML model is trained and deployed from EdgeImpulse framework for hand gesture recognition and based on the hand movements, Arduino Nano 33 BLE device having 6-axis I…
Peking University（北京大学）；Peng Cheng Laboratory（鹏城实验室）；Centre de Recherche Cerveau et Cognition (CERCO), UMR5549 CNRS - Univ. Toulouse 3（大脑与认知研究中心（CERCO），CNRS UMR5549 - 图卢兹第三大学）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Spiking Neural Networks (SNNs) have attracted enormous research interest due to temporal information processing capability, low power consumption, and high biological plausibility. However, the formulation of efficient and high-performance learning algorithms for SNNs is still challenging. Most existing learning methods learn weights only, and require manual tuning of the membrane-related parameters that determine the dynamics of a single spiking neuron. These parameters are typically chosen to be the same for all neurons, which limits the diversity of neurons and thus the expressiveness of th…
Nanjing University of Posts and Telecommunications（南京邮电大学）；University of Oulu（奥卢大学）；Stanford University（斯坦福大学）
Micro-expression recognition (MER) is valuable because micro-expressions (MEs) can reveal genuine emotions. Most works take image sequences as input and cannot effectively explore ME information because subtle ME-related motions are easily submerged in unrelated information. Instead, the facial landmark is a low-dimensional and compact modality, which achieves lower computational cost and potentially concentrates on ME-related movement features. However, the discriminability of facial landmarks for MER is unclear. Thus, this paper explores the contribution of facial landmarks and proposes a no…
K. J. Somaiya College of Engineering（K. J. 索马亚工程学院）；University of Mumbai（孟买大学）
This paper presents a system which can recognise hand poses & gestures from the Indian Sign Language (ISL) in real-time using grid-based features. This system attempts to bridge the communication gap between the hearing and speech impaired and the rest of the society. The existing solutions either provide relatively low accuracy or do not work in real-time. This system provides good results on both the parameters. It can identify 33 hand poses and some gestures from the ISL. Sign Language is captured from a smartphone camera and its frames are transmitted to a remote server for processing. The…
Live Demonstration: Neuromorphic Radar for Gesture Recognition
Department of Electronic Systems Engineering, Indian Institute of Science, Bangalore, India 560012（电子系统工程系，印度科学院，班加罗尔，印度560012）；Department of Electrical Engineering, Indian Institute of Science, Bangalore, India 560012（电气工程系，印度科学院，班加罗尔，印度560012）
Benchmarking Micro-action Recognition: Dataset, Methods, and Applications
Hefei University of Technology (HFUT)（合肥工业大学）；Hefei Comprehensive National Science Center, Institute of Artificial Intelligence（合肥综合性国家科学中心人工智能研究院）；Anhui Zhonghuitong Technology Co., Ltd.（安徽中汇通科技有限公司）；Lanzhou University（兰州大学）
