In this paper, a deep learning-based model for 3D human motion generation from the text is proposed via gesture action classification and an autoregressive model. The model focuses on generating special gestures that express human thinking, such as waving and nodding. To achieve the goal, the proposed method predicts expression from the sentences using a text classification model based on a pretrained language model and generates gestures using the gate recurrent unit-based autoregressive model. Especially, we proposed the loss for the embedding space for restoring raw motions and generating intermediate motions well. Moreover, the novel data augmentation method and stop token are proposed to generate variable length motions. To evaluate the text classification model and 3D human motion generation model, a gesture action classification dataset and action-based gesture dataset are collected. With several experiments, the proposed method successfully generates perceptually natural and realistic 3D human motion from the text. Moreover, we verified the effectiveness of the proposed method using a public-available action recognition dataset to evaluate cross-dataset generalization performance.
GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
University of Rochester（罗切斯特大学）；University of Tokyo（东京大学）；University of California Santa Cruz（加州大学圣克鲁兹分校）；University of California Los Angeles（加州大学洛杉矶分校）；Meta
Macau University of Science and Technology（澳门科技大学）；Communication University of Zhejiang（浙江传媒学院）；Ningbo University of Finance & Economics（宁波财经学院）
Speech-driven gesture synthesis is a field of growing interest in virtual human creation. However, a critical challenge is the inherent intricate one-to-many mapping between speech and gestures. Previous studies have explored and achieved significant progress with generative models. Notwithstanding, most synthetic gestures are still vastly less natural. This paper presents DiffMotion, a novel speech-driven gesture synthesis architecture based on diffusion models. The model comprises an autoregressive temporal encoder and a denoising diffusion probability Module. The encoder extracts the tempor…
MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantization
Communication University of China（中国传媒大学）；China Mobile Research Institute（中国移动研究院）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；The Chinese University of Hong Kong（香港中文大学）
Macau University of Science and Technology（澳门科技大学）；Communication University of Zhejiang（浙江传媒学院）；Zhejiang Lab（之江实验室）；Communication University of China（中国传媒大学）；Zhejiang University（浙江大学）；Zhejiang Institute of Economics and Trade（浙江经贸职业技术学院）
Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Model
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent AI Lab（腾讯AI实验室）；The Hong Kong University of Science and Technology（香港科技大学）
DiffVC: A Non-autoregressive Framework Based on Diffusion Model for Video Captioning
School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Beijing University Of Technology（北京工业大学）
ETRI（韩国电子通信研究院）；KAIST（韩国科学技术院）；University of Science and Technology（科学技术大学）
For human-like agents, including virtual avatars and social robots, making proper gestures while speaking is crucial in human--agent interaction. Co-speech gestures enhance interaction experiences and make the agents look alive. However, it is difficult to generate human-like gestures due to the lack of understanding of how people gesture. Data-driven approaches attempt to learn gesticulation skills from human demonstrations, but the ambiguous and individual nature of gestures hinders learning. In this paper, we present an automatic gesture generation model that uses the multimodal context of …
SRM Institute of Science and Technology（SRM科学技术学院）
The hand gestures are one of the typical methods used in sign language. It is very difficult for the hearing-impaired people to communicate with the world. This project presents a solution that will not only automatically recognize the hand gestures but will also convert it into speech and text output so that impaired person can easily communicate with normal people. A camera attached to computer will capture images of hand and the contour feature extraction is used to recognize the hand gestures of the person. Based on the recognized gestures, the recorded soundtrack will be played.
GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Engineering Research Center of Mixed Reality and Advanced Display（北京混合现实与先进显示工程研究中心）；Institute of Technology, Beijing, China（北京理工大学）
Institute for Creative Technologies, University of Southern California（南加州大学创意技术研究所）；Electronics and Telecommunications Research Institute（韩国电子通信研究院）
Synthesizing realistic co-speech gestures is an important and yet unsolved problem for creating believable motions that can drive a humanoid robot to interact and communicate with human users. Such capability will improve the impressions of the robots by human users and will find applications in education, training, and medical services. One challenge in learning the co-speech gesture model is that there may be multiple viable gesture motions for the same speech utterance. The deterministic regression methods can not resolve the conflicting samples and may produce over-smoothed or damped motio…
VarGes: Improving Variation in Co-Speech 3D Gesture Generation via StyleCLIPS
Communication University of China（中国传媒大学）；Samsung Research America（三星美国研究院）；Hainan International College, Communication University of China（中国传媒大学海南国际学院）；Beihang University（北京航空航天大学）；Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）
LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation
ShanghaiTech University（上海科技大学）；Tencent AI Lab（腾讯人工智能实验室）；Intellindust；INRIA（法国国家信息与自动化研究所）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程研究中心）；Shanghai Engineering Research Center of Energy Efficient and Custom AI IC（上海高效节能与定制人工智能芯片工程研究中心）
The University of Tokyo（东京大学）；Keio University（庆应义塾大学）；Huawei Technologies Japan K.K.（华为技术日本株式会社）；Japan Advanced Institute of Science and Technology（日本北陆先端科学技术大学院大学）；Huawei Turkey R&D Center（华为土耳其研发中心）
Achieving realistic, vivid, and human-like synthesized conversational gestures conditioned on multi-modal data is still an unsolved problem due to the lack of available datasets, models and standard evaluation metrics. To address this, we build Body-Expression-Audio-Text dataset, BEAT, which has i) 76 hours, high-quality, multi-modal data captured from 30 speakers talking with eight different emotions and in four different languages, ii) 32 millions frame-level emotion and semantic relevance annotations. Our statistical analysis on BEAT demonstrates the correlation of conversational gestures w…
When Semantics Matter: Reliability-Aware Semantic-Rhythm Control for Co-Speech Gesture Generation
Hainan International College, Communication University of China（中国传媒大学海南国际学院）；School of Data Science and Intelligent Media, Communication University of China（中国传媒大学数据科学与智能媒体学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间安全学院）
The Chinese University of Hong Kong（香港中文大学）；Monash University（莫纳什大学）；Zhejiang University（浙江大学）；SenseTime Research（商汤科技研究院）；Nanyang Technological University（南洋理工大学）
Generating speech-consistent body and gesture movements is a long-standing problem in virtual avatar creation. Previous studies often synthesize pose movement in a holistic manner, where poses of all joints are generated simultaneously. Such a straightforward pipeline fails to generate fine-grained co-speech gestures. One observation is that the hierarchical semantics in speech and the hierarchical structures of human gestures can be naturally described into multiple granularities and associated together. To fully utilize the rich connections between speech audio and human gestures, we propose…
SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning
Max Planck Institute for Psycholinguistics（马克斯·普朗克心理学研究所）；Donders Institute for Brain Cognition and Behaviour（多纳尔斯脑认知与行为研究所）；Utrecht University（乌得勒支大学）
HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generation
College of Mechanical and Electronic Engineering, Northwest A&F University（西北农林科技大学机械与电子工程学院）；Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学脑智能科学与技术研究院）；School of Data Science, Fudan University（复旦大学数据科学学院）；College of Information Engineering, Northwest A&F University（西北农林科技大学信息工程学院）
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tencent AI Lab（腾讯人工智能实验室）
People may perform diverse gestures affected by various mental and physical factors when speaking the same sentences. This inherent one-to-many relationship makes co-speech gesture generation from audio particularly challenging. Conventional CNNs/RNNs assume one-to-one mapping, and thus tend to predict the average of all possible target motions, easily resulting in plain/boring motions during inference. So we propose to explicitly model the one-to-many audio-to-motion mapping by splitting the cross-modal latent code into shared code and motion-specific code. The shared code is expected to be r…
Considerable effort has been devoted to the automatic extraction of information about action of the face from image sequences. Within the context of human-computer interaction (HCI) we may distinguish systems that allow expression from those which aim at recognition. Most of the work in facial action processing has been directed at automatically recognizing affect from facial actions. By contrast, facial gesture interfaces, which respond to deliberate facial actions, have received comparatively little attention. This paper reviews several projects on vision-based interfaces that rely on facial…
