Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates
ShanghaiTech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）；Shanghai Engineering Research Center of Energy Efficient and Custom AI IC（上海能效与定制AI集成电路工程技术研究中心）
Co-speech gesture generation is to synthesize a gesture sequence that not only looks real but also matches with the input speech audio. Our method generates the movements of a complete upper body, including arms, hands, and the head. Although recent data-driven methods achieve great success, challenges still exist, such as limited variety, poor fidelity, and lack of objective metrics. Motivated by the fact that the speech cannot fully determine the gesture, we design a method that learns a set of gesture template vectors to model the latent conditions, which relieve the ambiguity. For our method, the template vector determines the general appearance of a generated gesture sequence, while the speech audio drives subtle movements of the body, both indispensable for synthesizing a realistic gesture sequence. Due to the intractability of an objective metric for gesture-speech synchronization, we adopt the lip-sync error as a proxy metric to tune and evaluate the synchronization ability of our model. Extensive experiments show the superiority of our method in both objective and subjective evaluations on fidelity and synchronization.
Squiggle is a template-based glyph recognizer in the lineage of `$1 Recognizer' and `Protractor'. It seeks a good fit linear affine mapping between the input and template glyphs which are represented as a list of milestone points along the glyph path. The algorithm can recognize input glyphs invariant of rotation, scaling, skew, and reflection symmetries. In practice the algorithm is fast and robust enough to recognize user-generated glyphs as they are being drawn in real time, and to project `shadows' of the matching templates as feedback.
The use of hand gestures can be a useful tool for many applications in the human-computer interaction community. In a broad range of areas hand gesture techniques can be applied specifically in sign language recognition, robotic surgery, etc. In the process of hand gesture recognition, proper detection, and tracking of the moving hand become challenging due to the varied shape and size of the hand. Here the objective is to track the movement of the hand irrespective of the shape, size, and color of the hand. And, for this, a motion template guided by optical flow (OFMT) is proposed. OFMT is a …
The Chinese University of Hong Kong（香港中文大学）；Monash University（莫纳什大学）；Cornell University（康奈尔大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanyang Technological University（南洋理工大学）
Co-speech gesture is crucial for human-machine interaction and digital entertainment. While previous works mostly map speech audio to human skeletons (e.g., 2D keypoints), directly generating speakers' gestures in the image domain remains unsolved. In this work, we formally define and study this challenging problem of audio-driven co-speech gesture video generation, i.e., using a unified framework to generate speaker image sequence driven by speech audio. Our key insight is that the co-speech gestures can be decomposed into common motion patterns and subtle rhythmic dynamics. To this end, we p…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Adobe Research（奥多比研究院）
Human speech is often accompanied by body gestures including arm and hand gestures. We present a method that reenacts a high-quality video with gestures matching a target speech audio. The key idea of our method is to split and re-assemble clips from a reference video through a novel video motion graph encoding valid transitions between clips. To seamlessly connect different clips in the reenactment, we propose a pose-aware video blending network which synthesizes video frames around the stitched frames between two clips. Moreover, we developed an audio-based gesture searching algorithm to fin…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Facebook Reality Labs（Facebook现实实验室）
We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input. Our algorithm uses a CNN architecture that leverages the inherent correlation between facial expression and hand gestures. Synthesis of conversational body gestures is a multi-modal problem since many similar gestures can plausibly accompany the same input speech. To synthesize plausible body gestures in this setting, we train a Generative Adversarial Network (GAN) based model that …
American University（美利坚大学）；Graz University Of Technology（格拉茨技术大学）
We propose a real-time system for synthesizing gestures directly from speech. Our data-driven approach is based on Generative Adversarial Neural Networks to model the speech-gesture relationship. We utilize the large amount of speaker video data available online to train our 3D gesture model. Our model generates speaker-specific gestures by taking consecutive audio input chunks of two seconds in length. We animate the predicted gestures on a virtual avatar. We achieve a delay below three seconds between the time of audio input and gesture animation. Code and videos are available at https://git…
Macau University of Science and Technology（澳门科技大学）；Communication University of Zhejiang（浙江传媒学院）；Ningbo University of Finance & Economics（宁波财经学院）
Speech-driven gesture synthesis is a field of growing interest in virtual human creation. However, a critical challenge is the inherent intricate one-to-many mapping between speech and gestures. Previous studies have explored and achieved significant progress with generative models. Notwithstanding, most synthetic gestures are still vastly less natural. This paper presents DiffMotion, a novel speech-driven gesture synthesis architecture based on diffusion models. The model comprises an autoregressive temporal encoder and a denoising diffusion probability Module. The encoder extracts the tempor…
The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Animating virtual avatars to make co-speech gestures facilitates various applications in human-machine interaction. The existing methods mainly rely on generative adversarial networks (GANs), which typically suffer from notorious mode collapse and unstable training, thus making it difficult to learn accurate audio-gesture joint distributions. In this work, we propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture), to effectively capture the cross-modal audio-to-gesture associations and preserve temporal coherence for high-fidelity audio-driven co-speech gest…
A car driver knows how to react on the gestures of the traffic officers. Clearly, this is not the case for the autonomous vehicle, unless it has road traffic control gesture recognition functionalities. In this work, we address the limitation of the existing autonomous driving datasets to provide learning data for traffic control gesture recognition. We introduce a dataset that is based on 3D body skeleton input to perform traffic control gesture classification on every time step. Our dataset consists of 250 sequences from several actors, ranging from 16 to 90 seconds per sequence. To evaluate…
Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Model
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent AI Lab（腾讯AI实验室）；The Hong Kong University of Science and Technology（香港科技大学）
VarGes: Improving Variation in Co-Speech 3D Gesture Generation via StyleCLIPS
Communication University of China（中国传媒大学）；Samsung Research America（三星美国研究院）；Hainan International College, Communication University of China（中国传媒大学海南国际学院）；Beihang University（北京航空航天大学）；Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）
QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；The Chinese University of Hong Kong（香港中文大学）；Huawei Cloud Computing Technologies Co., Ltd（华为云计算技术有限公司）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Graz University of Technology（格拉茨技术大学）；American University（美利坚大学）
People communicate using both speech and non-verbal signals such as gestures, face expression or body pose. Non-verbal signals impact the meaning of the spoken utterance in an abundance of ways. An absence of non-verbal signals impoverishes the process of communication. Yet, when users are represented as avatars, it is difficult to translate non-verbal signals along with the speech into the virtual world without specialized motion-capture hardware. In this paper, we propose a novel, data-driven technique for generating gestures directly from speech. Our approach is based on the application of …
This paper presents a novel system that enables intelligent robots to exhibit realistic body gestures while communicating with humans. The proposed system consists of a listening model and a speaking model used in corresponding conversational phases. Both models are adapted from the sequence-to-sequence (seq2seq) architecture to synthesize body gestures represented by the movements of twelve upper-body keypoints. All the extracted 2D keypoints are firstly 3D-transformed, then rotated and normalized to discard irrelevant information. Substantial videos of human conversations from Youtube are co…
KTH Royal Institute of Technology（瑞典皇家理工学院）；Hokkai Gakuen University（北海学园大学）；Aoyama Gakuin University（青山学院大学）
This paper presents a novel framework for speech-driven gesture production, applicable to virtual agents to enhance human-computer interaction. Specifically, we extend recent deep-learning-based, data-driven methods for speech-driven gesture generation by incorporating representation learning. Our model takes speech as input and produces gestures as output, in the form of a sequence of 3D coordinates. We provide an analysis of different representations for the input (speech) and the output (motion) of the network by both objective and subjective evaluations. We also analyse the importance of s…
Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Baidu（百度）；Tokyo Institute of Technology（东京工业大学）；Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；Nanyang Technological University（南洋理工大学）
KTH Royal Institute of Technology（瑞典皇家理工学院）；University of California, Davis（加利福尼亚大学戴维斯分校）
We propose a new framework for gesture generation, aiming to allow data-driven approaches to produce more semantically rich gestures. Our approach first predicts whether to gesture, followed by a prediction of the gesture properties. Those properties are then used as conditioning for a modern probabilistic gesture-generation model capable of high-quality output. This empowers the approach to generate gestures that are both diverse and representational. Follow-ups and more information can be found on the project page: https://svito-zar.github.io/speech2properties2gestures/ .
Macau University of Science and Technology（澳门科技大学）；Communication University of Zhejiang（浙江传媒学院）；Zhejiang Lab（之江实验室）；Communication University of China（中国传媒大学）；Zhejiang University（浙江大学）；Zhejiang Institute of Economics and Trade（浙江经贸职业技术学院）
Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark
KTH Royal Institute of Technology（皇家理工学院）；Bielefeld University（比勒菲尔德大学）；University of Science – VNUHCM（越南胡志明市国家大学）；Independent Researcher（独立研究者）；Motorica AB（Motorica AB公司）；Peking University（北京大学）；Tsinghua University（清华大学）；Astribot（Astribot公司）；Max-Planck Institute for Informatics, SIC（马克斯·普朗克信息研究所，SIC）；Trinity College Dublin（都柏林大学）；University of California, Davis（加州大学戴维斯分校）；SEED – Electronic Arts（SEED–电子艺界）；Electronics and Telecommunications Research Institute (ETRI)（电子电信研究院）
Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Experts
Department of Automation, Tsinghua University（自动化系，清华大学）；Bytedance Inc.（字节跳动公司）；State Key Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播国家重点实验室，中国传媒大学）；School of Artificial Intelligence, Beijing Normal University（人工智能学院，北京师范大学）
The GENEA Challenge 2026: A Large-Scale Disentangled Evaluation of Speech-Driven Gesture Generation on the Seamless Interaction Dataset
KTH Royal Institute of Technology（瑞典皇家理工学院）；Bielefeld University（比勒费尔德大学）；National Library of Sweden（瑞典国家图书馆）；Electronics and Telecommunications Research Institute (ETRI)（电子通信研究院）；Motorica AB（Motorica AB公司）
ETRI（韩国电子通信研究院）；KAIST（韩国科学技术院）；University of Science and Technology（科学技术大学）
For human-like agents, including virtual avatars and social robots, making proper gestures while speaking is crucial in human--agent interaction. Co-speech gestures enhance interaction experiences and make the agents look alive. However, it is difficult to generate human-like gestures due to the lack of understanding of how people gesture. Data-driven approaches attempt to learn gesticulation skills from human demonstrations, but the ambiguous and individual nature of gestures hinders learning. In this paper, we present an automatic gesture generation model that uses the multimodal context of …
Institute for Creative Technologies, University of Southern California（南加州大学创意技术研究所）；Electronics and Telecommunications Research Institute（韩国电子通信研究院）
Synthesizing realistic co-speech gestures is an important and yet unsolved problem for creating believable motions that can drive a humanoid robot to interact and communicate with human users. Such capability will improve the impressions of the robots by human users and will find applications in education, training, and medical services. One challenge in learning the co-speech gesture model is that there may be multiple viable gesture motions for the same speech utterance. The deterministic regression methods can not resolve the conflicting samples and may produce over-smoothed or damped motio…
Trinity College Dublin（都柏林圣三一学院）；University of California Davis（加利福尼亚大学戴维斯分校）
Gesture behavior is a natural part of human conversation. Much work has focused on removing the need for tedious hand-animation to create embodied conversational agents by designing speech-driven gesture generators. However, these generators often work in a black-box manner, assuming a general relationship between input speech and output motion. As their success remains limited, we investigate in more detail how speech may relate to different aspects of gesture motion. We determine a number of parameters characterizing gesture, such as speed and gesture size, and explore their relationship to …
End to End AI System for Surgical Gesture Sequence Recognition and Clinical Outcome Prediction
Department of Computational Biomedicine, Center for Artificial Intelligence Research and Education（计算生物医学系，人工智能研究与教育中心）；Cedars Sinai Medical Center（Cedars Sinai 医疗中心）
Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；The Chinese University of Hong Kong（香港中文大学）；Huawei Cloud Computing Technologies Co., Ltd（华为云计算技术有限公司）
RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction
Tsinghua University（清华大学）；Galbot Inc.（Galbot公司）；Beijing Institute of Technology（北京理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Peking University（北京大学）；Shanghai Qi Zhi Institute（上海智知研究院）
AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars
Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究院（TeleAI）、中国电信）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（未来区块链与隐私计算先进创新中心）；School of Artificial Intelligence, Beihang University（北航人工智能学院）；Hangzhou International Innovation Institute, Beihang University（杭州国际创新研究院，北航）；School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）；GigaAI（极佳科技）
Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis
KTH Royal Institute of Technology（瑞典皇家理工学院）；Bielefeld University（比勒费尔德大学）；ETRI（韩国电子通信研究院）；SEED – Electronic Arts（艺电SEED工作室）；Motorica AB（莫托里卡公司）；University of Science – VNUHCM（胡志明市国家大学科学大学）；Trinity College Dublin（都柏林圣三一学院）；University of California, Davis（加州大学戴维斯分校）
