Generating vivid and diverse 3D co-speech gestures is crucial for various applications in animating virtual avatars. While most existing methods can generate gestures from audio directly, they usually overlook that emotion is one of the key factors of authentic co-speech gesture generation. In this work, we propose EmotionGesture, a novel framework for synthesizing vivid and diverse emotional co-speech 3D gestures from audio. Considering emotion is often entangled with the rhythmic beat in speech audio, we first develop an Emotion-Beat Mining module (EBM) to extract the emotion and audio beat features as well as model their correlation via a transcript-based visual-rhythm alignment. Then, we propose an initial pose based Spatial-Temporal Prompter (STP) to generate future gestures from the given initial poses. STP effectively models the spatial-temporal correlations between the initial poses and the future gestures, thus producing the spatial-temporal coherent pose prompt. Once we obtain pose prompts, emotion, and audio beat features, we will generate 3D co-speech gestures through a transformer architecture. However, considering the poses of existing datasets often contain jittering effects, this would lead to generating unstable gestures. To address this issue, we propose an effective objective function, dubbed Motion-Smooth Loss. Specifically, we model motion offset to compensate for jittering ground-truth by forcing gestures to be smooth. Last, we present an emotion-conditioned VAE to sample emotion features, enabling us to generate diverse emotional results. Extensive experiments demonstrate that our framework outperforms the state-of-the-art, achieving vivid and diverse emotional co-speech 3D gestures. Our code and dataset will be released at the project page: https://xingqunqi-lab.github.io/Emotion-Gesture-Web/
The University of Tokyo（东京大学）；Keio University（庆应义塾大学）；Huawei Technologies Japan K.K.（华为技术日本株式会社）；Japan Advanced Institute of Science and Technology（日本北陆先端科学技术大学院大学）；Huawei Turkey R&D Center（华为土耳其研发中心）
Achieving realistic, vivid, and human-like synthesized conversational gestures conditioned on multi-modal data is still an unsolved problem due to the lack of available datasets, models and standard evaluation metrics. To address this, we build Body-Expression-Audio-Text dataset, BEAT, which has i) 76 hours, high-quality, multi-modal data captured from 30 speakers talking with eight different emotions and in four different languages, ii) 32 millions frame-level emotion and semantic relevance annotations. Our statistical analysis on BEAT demonstrates the correlation of conversational gestures w…
We introduce a new dataset for the emotional artificial intelligence research: identity-free video dataset for Micro-Gesture Understanding and Emotion analysis (iMiGUE). Different from existing public datasets, iMiGUE focuses on nonverbal body gestures without using any identity information, while the predominant researches of emotion analysis concern sensitive biometric data, like face and speech. Most importantly, iMiGUE focuses on micro-gestures, i.e., unintentional behaviors driven by inner feelings, which are different from ordinary scope of gestures from other gesture datasets which are …
Identity-free Artificial Emotional Intelligence via Micro-Gesture Understanding
Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学（LUT））；Great Bay University（大湾区大学）；Imperial College London（伦敦帝国理工学院）；Technical University of Munich（慕尼黑工业大学）
University of Tartu（塔尔图大学）；Lodz University of Technology（罗兹理工大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Hasan Kalyoncu University（哈桑·卡利永丘大学）
Automatic emotion recognition has become a trending research topic in the past decade. While works based on facial expressions or speech abound, recognizing affect from body gestures remains a less explored topic. We present a new comprehensive survey hoping to boost research in the field. We first introduce emotional body gestures as a component of what is commonly known as "body language" and comment general aspects as gender differences and culture dependence. We then define a complete framework for automatic emotional body gesture recognition. We introduce person detection and comment stat…
Active Inference for Micro-Gesture Recognition: EFE-Guided Temporal Sampling and Adaptive Learning
Tianjin Normal University（天津师范大学）；Shenzhen University（深圳大学）；The State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；School of New Media and Communications, Tianjin University（天津大学新媒体与传播学院）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Psychology, Chinese Academy of Sciences（中国科学院心理研究所）
Although automatic emotion recognition from facial expressions and speech has made remarkable progress, emotion recognition from body gestures has not been thoroughly explored. People often use a variety of body language to express emotions, and it is difficult to enumerate all emotional body gestures and collect enough samples for each category. Therefore, recognizing new emotional body gestures is critical for better understanding human emotions. However, the existing methods fail to accurately determine which emotional state a new body gesture belongs to. In order to solve this problem, we …
Powder AI Research（Powder AI 研究院）；Université Paris-Est, LISSI, UPEC（巴黎东大学 LISSI 实验室 / 巴黎东克雷泰伊大学）
Emotional expressions are the behaviors that communicate our emotional state or attitude to others. They are expressed through verbal and non-verbal communication. Complex human behavior can be understood by studying physical features from multiple modalities; mainly facial, vocal and physical gestures. Recently, spontaneous multi-modal emotion recognition has been extensively studied for human behavior analysis. In this paper, we propose a new deep learning-based approach for audio-visual emotion recognition. Our approach leverages recent advances in deep learning like knowledge distillation …
Multi-Track Multimodal Learning on iMiGUE: Micro-Gesture and Emotion Recognition
Russian - Armenian University（俄罗斯-亚美尼亚大学）；Sber AI Lab（Sber AI实验室）；HSE University（俄罗斯高等经济大学）；Innopolis University（Innopolis大学）；ISP RAS Research Center for Trusted Artificial Intelligence（俄罗斯科学院可信人工智能研究中心）
We present a novel generalized zero-shot algorithm to recognize perceived emotions from gestures. Our task is to map gestures to novel emotion categories not encountered in training. We introduce an adversarial, autoencoder-based representation learning that correlates 3D motion-captured gesture sequence with the vectorized representation of the natural-language perceived emotion terms using word2vec embeddings. The language-semantic embedding provides a representation of the emotion label space, and we leverage this underlying distribution to map the gesture-sequences to the appropriate categ…
In the context of education technology, empathic interaction with the user and feedback by the learning system using multiple inputs such as video, voice and text inputs is an important area of research. In this paper, a nonintrusive, standalone model for intelligent assessment of alertness and emotional state as well as generation of appropriate feedback has been proposed. Using the non-intrusive visual cues, the system classifies emotion and alertness state of the user, and provides appropriate feedback according to the detected cognitive state using facial expressions, ocular parameters, po…
Macau University of Science and Technology（澳门科技大学）；Communication University of Zhejiang（浙江传媒学院）；Zhejiang Lab（之江实验室）；Communication University of China（中国传媒大学）；Zhejiang University（浙江大学）；Zhejiang Institute of Economics and Trade（浙江经贸职业技术学院）
InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Meituan New Laboratory of Pattern Recognition (NLPR), CASIA（美团模式识别新实验室（NLPR），中国科学院自动化所）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Meituan Division of AMC and Department of ECE, HKUST（美团AMC部门和香港科技大学电子与计算机工程系）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中国科学院自动化所）；Meituan School of Artificial Intelligence, University of Chinese Academy of Sciences（美团人工智能学院，中国科学院大学）；Meituan（美团）
