This work addresses the problem of generating 3D holistic body motions from human speech. Given a speech recording, we synthesize sequences of 3D body poses, hand gestures, and facial expressions that are realistic and diverse. To achieve this, we first build a high-quality dataset of 3D holistic body meshes with synchronous speech. We then define a novel speech-to-motion generation framework in which the face, body, and hands are modeled separately. The separated modeling stems from the fact that face articulation strongly correlates with human speech, while body poses and hand gestures are less correlated. Specifically, we employ an autoencoder for face motions, and a compositional vector-quantized variational autoencoder (VQ-VAE) for the body and hand motions. The compositional VQ-VAE is key to generating diverse results. Additionally, we propose a cross-conditional autoregressive model that generates body poses and hand gestures, leading to coherent and realistic motions. Extensive experiments and user studies demonstrate that our proposed approach achieves state-of-the-art performance both qualitatively and quantitatively. Our novel dataset and code will be released for research purposes at https://talkshow.is.tue.mpg.de.
Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation
Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所）；Utrecht University（乌得勒支大学）；Donders Institute for Brain Cognition and Behaviour（唐德斯脑认知与行为研究所）
IIT Kanpur（印度理工学院坎普尔分校）；IIIT Hyderabad（印度海得拉巴国际信息技术学院）；University of Bath（巴斯大学）
We aim to solve the highly challenging task of generating continuous sign language videos solely from speech segments for the first time. Recent efforts in this space have focused on generating such videos from human-annotated text transcripts without considering other modalities. However, replacing speech with sign language proves to be a practical solution while communicating with people suffering from hearing loss. Therefore, we eliminate the need of using text as input and design techniques that work for more natural, continuous, freely uttered speech covering an extensive vocabulary. Sinc…
EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；JAIST（日本先进科学技术研究所）；Tsinghua University（清华大学）；The University of Tokyo（东京大学）；Keio University（庆应义塾大学）
Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony
Wolf 1069B Lab, Sany Group（沃尔夫1069B实验室，三一集团）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Engineering and Technology, University of Washington, Tacoma（华盛顿大学塔科姆分校工程与科技学院）；Alibaba Group（阿里巴巴集团）；Department of Engineering, King’s College London（伦敦大学国王学院工程系）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）
3DiFACE: Synthesizing and Editing Holistic 3D Facial Animation
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck ETH Center for Intelligent Systems（马克斯·普朗克-苏黎世联邦理工学院智能系统中心）；Technical University of Darmstadt（德累斯顿技术大学）；ETH Zürich（苏黎世联邦理工学院）；Microsoft Mixed Reality & AI Lab, UK（英国微软混合现实与人工智能实验室）
本文提出 3DiFACE，利用全卷积扩散、说话风格个性化与稀疏引导运动扩散，从单一音频生成并可关键帧编辑多样且真实的整体 3D 面部动画。
KTH Royal Institute of Technology（瑞典皇家理工学院）；University of California, Davis（加利福尼亚大学戴维斯分校）
We propose a new framework for gesture generation, aiming to allow data-driven approaches to produce more semantically rich gestures. Our approach first predicts whether to gesture, followed by a prediction of the gesture properties. Those properties are then used as conditioning for a modern probabilistic gesture-generation model capable of high-quality output. This empowers the approach to generate gestures that are both diverse and representational. Follow-ups and more information can be found on the project page: https://svito-zar.github.io/speech2properties2gestures/ .
Spatial relationships between objects provide important information for text-based image retrieval. As users are more likely to describe a scene from a real world perspective, using 3D spatial relationships rather than 2D relationships that assume a particular viewing direction, one of the main challenges is to infer the 3D structure that bridges images with users' text descriptions. However, direct inference of 3D structure from images requires learning from large scale annotated data. Since interactions between objects can be reduced to a limited set of atomic spatial relations in 3D, we stu…
SpeechAct: Towards Generating Whole-body Motion from Speech
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Department of Automation, Tsinghua University（清华大学自动化系）；NNKosmos Technology（NNKosmos科技公司）
We propose an end to end deep learning approach for generating real-time facial animation from just audio. Specifically, our deep architecture employs deep bidirectional long short-term memory network and attention mechanism to discover the latent representations of time-varying contextual information within the speech and recognize the significance of different information contributed to certain face status. Therefore, our model is able to drive different levels of facial movements at inference and automatically keep up with the corresponding pitch and latent speaking style in the input audio…
The Chinese University of Hong Kong（香港中文大学）；Monash University（莫纳什大学）；Zhejiang University（浙江大学）；SenseTime Research（商汤科技研究院）；Nanyang Technological University（南洋理工大学）
Generating speech-consistent body and gesture movements is a long-standing problem in virtual avatar creation. Previous studies often synthesize pose movement in a holistic manner, where poses of all joints are generated simultaneously. Such a straightforward pipeline fails to generate fine-grained co-speech gestures. One observation is that the hierarchical semantics in speech and the hierarchical structures of human gestures can be naturally described into multiple granularities and associated together. To fully utilize the rich connections between speech audio and human gestures, we propose…
Recent adoption of deep learning methods to the field of machine lipreading research gives us two options to pursue to improve system performance. Either, we develop end-to-end systems holistically or, we experiment to further our understanding of the visual speech signal. The latter option is more difficult but this knowledge would enable researchers to both improve systems and apply the new knowledge to other domains such as speech therapy. One challenge in lipreading systems is the correct labeling of the classifiers. These labels map an estimated function between visemes on the lips and th…
The presence of a corresponding talking face has been shown to significantly improve speech intelligibility in noisy conditions and for hearing impaired population. In this paper, we present a system that can generate landmark points of a talking face from an acoustic speech in real time. The system uses a long short-term memory (LSTM) network and is trained on frontal videos of 27 different speakers with automatically extracted face landmarks. After training, it can produce talking face landmarks from the acoustic speech of unseen speakers and utterances. The training phase contains three key…
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tencent AI Lab（腾讯AI实验室）
Generating conversational gestures from speech audio is challenging due to the inherent one-to-many mapping between audio and body motions. Conventional CNNs/RNNs assume one-to-one mapping, and thus tend to predict the average of all possible target motions, resulting in plain/boring motions during inference. In order to overcome this problem, we propose a novel conditional variational autoencoder (VAE) that explicitly models one-to-many audio-to-motion mapping by splitting the cross-modal latent code into shared code and motion-specific code. The shared code mainly models the strong correlati…
Generating realistic lip motion from audio to simulate speech production is critical for driving natural character animation. Previous research has shown that traditional metrics used to optimize and assess models for generating lip motion from speech are not a good indicator of subjective opinion of animation quality. Devising metrics that align with subjective opinion first requires understanding what impacts human perception of quality. In this work, we focus on the degree of articulation and run a series of experiments to study how articulation strength impacts human perception of lip moti…
Holistic-Motion2D: Scalable Whole-body Human Motion Generation in 2D Space
Tsinghua University（清华大学）；The University of Sydney（悉尼大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Alibaba Group（阿里巴巴集团）；HKUST（香港科技大学）；Shandong University（山东大学）
Speech Audio Generation from dynamic MRI via a Knowledge Enhanced Conditional Variational Autoencoder
Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；School of Software Engineering, Xi'an Jiaotong University（西安交通大学软件工程学院）；Wake Forest University School of Medicine（威克森林大学医学学院）；Department of Radiology, Harvard Medical School（哈佛医学院放射科）
Sharif University of Technology（谢里夫理工大学）；imec（imec研究所）；Ghent University（根特大学）
We present a novel approach to generating photo-realistic images of a face with accurate lip sync, given an audio input. By using a recurrent neural network, we achieved mouth landmarks based on audio features. We exploited the power of conditional generative adversarial networks to produce highly-realistic face conditioned on a set of landmarks. These two networks together are capable of producing a sequence of natural faces in sync with an input audio track.
Northwestern Polytechnical University（西北工业大学）；Delft University of Technology（代尔夫特理工大学）
Automatically generating videos in which synthesized speech is synchronized with lip movements in a talking head has great potential in many human-computer interaction scenarios. In this paper, we present an automatic method to generate synchronized speech and talking-head videos on the basis of text and a single face image of an arbitrary person as input. In contrast to previous text-driven talking head generation methods, which can only synthesize the voice of a specific person, the proposed method is capable of synthesizing speech for any person that is inaccessible in the training stage. S…
