Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation
KTH Royal Institute of Technology（瑞典皇家理工学院）；Hokkai Gakuen University（北海学园大学）；Aoyama Gakuin University（青山学院大学）
Taras Kucherenko, Dai Hasegawa, Naoshi Kaneko, Gustav Eje Henter, Hedvig Kjellström
备注：
Extension of our IVA'19 paper. Accepted at the International Journal of Human-Computer Interaction. See more at https://svito-zar.github.io/audio2gestures/. arXiv admin note: substantial text overlap with arXiv:1903.03369
This paper presents a novel framework for speech-driven gesture production, applicable to virtual agents to enhance human-computer interaction. Specifically, we extend recent deep-learning-based, data-driven methods for speech-driven gesture generation by incorporating representation learning. Our model takes speech as input and produces gestures as output, in the form of a sequence of 3D coordinates. We provide an analysis of different representations for the input (speech) and the output (motion) of the network by both objective and subjective evaluations. We also analyse the importance of smoothing of the produced motion. Our results indicated that the proposed method improved on our baseline in terms of objective measures. For example, it better captured the motion dynamics and better matched the motion-speed distribution. Moreover, we performed user studies on two different datasets. The studies confirmed that our proposed method is perceived as more natural than the baseline, although the difference in the studies was eliminated by appropriate post-processing: hip-centering and smoothing. We conclude that it is important to take both motion representation and post-processing into account when designing an automatic gesture-production method.
KTH Royal Institute of Technology（瑞典皇家理工学院）；University of California, Davis（加利福尼亚大学戴维斯分校）
We propose a new framework for gesture generation, aiming to allow data-driven approaches to produce more semantically rich gestures. Our approach first predicts whether to gesture, followed by a prediction of the gesture properties. Those properties are then used as conditioning for a modern probabilistic gesture-generation model capable of high-quality output. This empowers the approach to generate gestures that are both diverse and representational. Follow-ups and more information can be found on the project page: https://svito-zar.github.io/speech2properties2gestures/ .
Trinity College Dublin（都柏林圣三一学院）；University of California Davis（加利福尼亚大学戴维斯分校）
Gesture behavior is a natural part of human conversation. Much work has focused on removing the need for tedious hand-animation to create embodied conversational agents by designing speech-driven gesture generators. However, these generators often work in a black-box manner, assuming a general relationship between input speech and output motion. As their success remains limited, we investigate in more detail how speech may relate to different aspects of gesture motion. We determine a number of parameters characterizing gesture, such as speed and gesture size, and explore their relationship to …
ETRI（韩国电子通信研究院）；KAIST（韩国科学技术院）；University of Science and Technology（科学技术大学）
For human-like agents, including virtual avatars and social robots, making proper gestures while speaking is crucial in human--agent interaction. Co-speech gestures enhance interaction experiences and make the agents look alive. However, it is difficult to generate human-like gestures due to the lack of understanding of how people gesture. Data-driven approaches attempt to learn gesticulation skills from human demonstrations, but the ambiguous and individual nature of gestures hinders learning. In this paper, we present an automatic gesture generation model that uses the multimodal context of …
GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
University of Rochester（罗切斯特大学）；University of Tokyo（东京大学）；University of California Santa Cruz（加州大学圣克鲁兹分校）；University of California Los Angeles（加州大学洛杉矶分校）；Meta
Macau University of Science and Technology（澳门科技大学）；Communication University of Zhejiang（浙江传媒学院）；Ningbo University of Finance & Economics（宁波财经学院）
Speech-driven gesture synthesis is a field of growing interest in virtual human creation. However, a critical challenge is the inherent intricate one-to-many mapping between speech and gestures. Previous studies have explored and achieved significant progress with generative models. Notwithstanding, most synthetic gestures are still vastly less natural. This paper presents DiffMotion, a novel speech-driven gesture synthesis architecture based on diffusion models. The model comprises an autoregressive temporal encoder and a denoising diffusion probability Module. The encoder extracts the tempor…
Institute for Creative Technologies, University of Southern California（南加州大学创意技术研究所）；Electronics and Telecommunications Research Institute（韩国电子通信研究院）
Synthesizing realistic co-speech gestures is an important and yet unsolved problem for creating believable motions that can drive a humanoid robot to interact and communicate with human users. Such capability will improve the impressions of the robots by human users and will find applications in education, training, and medical services. One challenge in learning the co-speech gesture model is that there may be multiple viable gesture motions for the same speech utterance. The deterministic regression methods can not resolve the conflicting samples and may produce over-smoothed or damped motio…
QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；The Chinese University of Hong Kong（香港中文大学）；Huawei Cloud Computing Technologies Co., Ltd（华为云计算技术有限公司）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
ShanghaiTech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）；Shanghai Engineering Research Center of Energy Efficient and Custom AI IC（上海能效与定制AI集成电路工程技术研究中心）
Co-speech gesture generation is to synthesize a gesture sequence that not only looks real but also matches with the input speech audio. Our method generates the movements of a complete upper body, including arms, hands, and the head. Although recent data-driven methods achieve great success, challenges still exist, such as limited variety, poor fidelity, and lack of objective metrics. Motivated by the fact that the speech cannot fully determine the gesture, we design a method that learns a set of gesture template vectors to model the latent conditions, which relieve the ambiguity. For our meth…
American University（美利坚大学）；Graz University Of Technology（格拉茨技术大学）
We propose a real-time system for synthesizing gestures directly from speech. Our data-driven approach is based on Generative Adversarial Neural Networks to model the speech-gesture relationship. We utilize the large amount of speaker video data available online to train our 3D gesture model. Our model generates speaker-specific gestures by taking consecutive audio input chunks of two seconds in length. We animate the predicted gestures on a virtual avatar. We achieve a delay below three seconds between the time of audio input and gesture animation. Code and videos are available at https://git…
The Chinese University of Hong Kong（香港中文大学）；Monash University（莫纳什大学）；Cornell University（康奈尔大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanyang Technological University（南洋理工大学）
Co-speech gesture is crucial for human-machine interaction and digital entertainment. While previous works mostly map speech audio to human skeletons (e.g., 2D keypoints), directly generating speakers' gestures in the image domain remains unsolved. In this work, we formally define and study this challenging problem of audio-driven co-speech gesture video generation, i.e., using a unified framework to generate speaker image sequence driven by speech audio. Our key insight is that the co-speech gestures can be decomposed into common motion patterns and subtle rhythmic dynamics. To this end, we p…
National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Bremen（不来梅大学）
Speaker extraction seeks to extract the clean speech of a target speaker from a multi-talker mixture speech. There have been studies to use a pre-recorded speech sample or face image of the target speaker as the speaker cue. In human communication, co-speech gestures that are naturally timed with speech also contribute to speech perception. In this work, we explore the use of co-speech gestures sequence, e.g. hand and body movements, as the speaker cue for speaker extraction, which could be easily obtained from low-resolution video recordings, thus more available than face recordings. We propo…
Leveraging Speech for Gesture Detection in Multimodal Communication
University of Amsterdam（阿姆斯特丹大学）；ScaDS.AI Dresden/Leipzig（德累斯顿/莱比锡ScaDS.AI研究院）；TU Dresden（德累斯顿工业大学）；Meertens Institute（梅滕斯研究所）；Radboud University（拉德堡德大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；University of Oxford（牛津大学）；Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所）
UC Berkeley（加州大学伯克利分校）；Zebra Medical Vision；MIT（麻省理工学院）
Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-modal translation from "in-the-wild'' monologue speech of a single speaker to their hand and arm motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and s…
The Chinese University of Hong Kong（香港中文大学）；Monash University（莫纳什大学）；Zhejiang University（浙江大学）；SenseTime Research（商汤科技研究院）；Nanyang Technological University（南洋理工大学）
Generating speech-consistent body and gesture movements is a long-standing problem in virtual avatar creation. Previous studies often synthesize pose movement in a holistic manner, where poses of all joints are generated simultaneously. Such a straightforward pipeline fails to generate fine-grained co-speech gestures. One observation is that the hierarchical semantics in speech and the hierarchical structures of human gestures can be naturally described into multiple granularities and associated together. To fully utilize the rich connections between speech audio and human gestures, we propose…
Graz University of Technology（格拉茨技术大学）；American University（美利坚大学）
People communicate using both speech and non-verbal signals such as gestures, face expression or body pose. Non-verbal signals impact the meaning of the spoken utterance in an abundance of ways. An absence of non-verbal signals impoverishes the process of communication. Yet, when users are represented as avatars, it is difficult to translate non-verbal signals along with the speech into the virtual world without specialized motion-capture hardware. In this paper, we propose a novel, data-driven technique for generating gestures directly from speech. Our approach is based on the application of …
SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning
Max Planck Institute for Psycholinguistics（马克斯·普朗克心理学研究所）；Donders Institute for Brain Cognition and Behaviour（多纳尔斯脑认知与行为研究所）；Utrecht University（乌得勒支大学）
Nitte Meenakshi Institute of Technology（尼特·米纳克希技术学院）
In this article gesture recognition and speech recognition applications are implemented on embedded systems with Tiny Machine Learning (TinyML). It features 3-axis accelerometer, 3-axis gyroscope and 3-axis magnetometer. The gesture recognition,provides an innovative approach nonverbal communication. It has wide applications in human-computer interaction and sign language. Here in the implementation of hand gesture recognition, TinyML model is trained and deployed from EdgeImpulse framework for hand gesture recognition and based on the hand movements, Arduino Nano 33 BLE device having 6-axis I…
GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Engineering Research Center of Mixed Reality and Advanced Display（北京混合现实与先进显示工程研究中心）；Institute of Technology, Beijing, China（北京理工大学）
BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer
Guangdong Academy of Sciences（广东省科学院）；The University of Hong Kong（香港大学）；The University of Edinburgh（爱丁堡大学）；Meta Reality Labs Research（Meta现实实验室）；National Institute of Informatics（国立信息学研究所）；Tohoku University（东北大学）
LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation
ShanghaiTech University（上海科技大学）；Tencent AI Lab（腾讯人工智能实验室）；Intellindust；INRIA（法国国家信息与自动化研究所）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程研究中心）；Shanghai Engineering Research Center of Energy Efficient and Custom AI IC（上海高效节能与定制人工智能芯片工程研究中心）
How can we teach robots or virtual assistants to gesture naturally? Can we go further and adapt the gesturing style to follow a specific speaker? Gestures that are naturally timed with corresponding speech during human communication are called co-speech gestures. A key challenge, called gesture style transfer, is to learn a model that generates these gestures for a speaking agent 'A' in the gesturing style of a target speaker 'B'. A secondary goal is to simultaneously learn to generate co-speech gestures for multiple speakers while remembering what is unique about each speaker. We call this ch…
Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；The Chinese University of Hong Kong（香港中文大学）；Huawei Cloud Computing Technologies Co., Ltd（华为云计算技术有限公司）
Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation
University of Amsterdam（阿姆斯特丹大学）；Maastricht University（马斯特里赫特大学）；Radboud University（拉德堡德大学）；Meertens Institute（米滕斯研究所）；MPI for Psycholinguistics（马克斯·普朗克心理语言学研究所）
SRM Institute of Science and Technology（SRM科学技术学院）
The hand gestures are one of the typical methods used in sign language. It is very difficult for the hearing-impaired people to communicate with the world. This project presents a solution that will not only automatically recognize the hand gestures but will also convert it into speech and text output so that impaired person can easily communicate with normal people. A camera attached to computer will capture images of hand and the contour feature extraction is used to recognize the hand gestures of the person. Based on the recognized gestures, the recorded soundtrack will be played.
I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue
Multimodal Language Department, Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所多模态语言部门）；Donders Institute for Brain, Cognition and Behaviour, Radboud University（拉德堡德大学大脑、认知与行为研究所）；Department of Advanced Computing Sciences, Maastricht University（马斯特里赫特大学高级计算科学系）；Institute for Logic, Language and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）
EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；JAIST（日本先进科学技术研究所）；Tsinghua University（清华大学）；The University of Tokyo（东京大学）；Keio University（庆应义塾大学）
The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Animating virtual avatars to make co-speech gestures facilitates various applications in human-machine interaction. The existing methods mainly rely on generative adversarial networks (GANs), which typically suffer from notorious mode collapse and unstable training, thus making it difficult to learn accurate audio-gesture joint distributions. In this work, we propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture), to effectively capture the cross-modal audio-to-gesture associations and preserve temporal coherence for high-fidelity audio-driven co-speech gest…
MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantization
Communication University of China（中国传媒大学）；China Mobile Research Institute（中国移动研究院）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；The Chinese University of Hong Kong（香港中文大学）
