Text-guided human body animation has advanced rapidly, yet facial animation lags due to the scarcity of well-annotated, text-paired facial corpora. To close this gap, we leverage foundation generative models to synthesize a large, balanced corpus of facial behavior. We design prompts suite covering emotions and head motions, generate about 80 hours of facial videos with multiple generators, and fit per-frame 3D facial parameters, yielding large-scale (prompt and parameter) pairs for training. Building on this dataset, we probe language models for bidirectional competence over facial motion via two complementary tasks: (1) Motion2Language: given a sequence of 3D facial parameters, the model produces natural-language descriptions capturing content, style, and dynamics; and (2) Language2Motion: given a prompt, the model synthesizes the corresponding sequence of 3D facial parameters via quantized motion tokens for downstream animation. Extensive experiments show that in this setting language models can both interpret and synthesize facial motion with strong generalization. To best of our knowledge, this is the first work to cast facial-parameter modeling as a language problem, establishing a unified path for text-conditioned facial animation and motion understanding.
The University of Sydney（悉尼大学）；Wuhan University（武汉大学）；AntGroup（蚂蚁集团）
The DeepFakes, which are the facial manipulation techniques, is the emerging threat to digital society. Various DeepFake detection methods and datasets are proposed for detecting such data, especially for face-swapping. However, recent researches less consider facial animation, which is also important in the DeepFake attack side. It tries to animate a face image with actions provided by a driving video, which also leads to a concern about the security of recent payment systems that reply on liveness detection to authenticate real users via recognising a sequence of user facial actions. However…
FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferences
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；USC-SJTU Institute of Cultural and Creative Industry, Shanghai Jiao Tong University（上海交通大学 USC-文化创意产业学院）；Institut Universitaire de France (IUF)（法国大学研究院）；University of Nantes（南特大学）
University of Chinese Academy of Sciences（中国科学院大学）
Nearly all existing Facial Action Coding System-based datasets that include facial action unit (AU) intensity information annotate the intensity values hierarchically using A--E levels. However, facial expressions change continuously and shift smoothly from one state to another. Therefore, it is more effective to regress the intensity value of local facial AUs to represent whole facial expression changes, particularly in the fields of expression transfer and facial animation. We introduce an extension of FEAFA in combination with the relabeled DISFA database, which is available at https://www.…
University of Chinese Academy of Sciences（中国科学院大学）；Capital Normal University（首都师范大学）
Facial expression analysis based on machine learning requires large number of well-annotated data to reflect different changes in facial motion. Publicly available datasets truly help to accelerate research in this area by providing a benchmark resource, but all of these datasets, to the best of our knowledge, are limited to rough annotations for action units, including only their absence, presence, or a five-level intensity according to the Facial Action Coding System. To meet the need for videos labeled in great detail, we present a well-annotated dataset named FEAFA for Facial Expression An…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural network on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Character Animation) takes any speech signal as input - even speech in languages other than English - and …
TalkingEyes: Pluralistic Speech-Driven 3D Eye Gaze Animation
School of Informatics, Xiamen University（厦门大学信息学院）；China Mobile (Hangzhou) Information Technology Co., Ltd.（中国移动（杭州）信息技术有限公司）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
This paper presents FaceXHuBERT, a text-less speech-driven 3D facial animation generation method that allows to capture personalized and subtle cues in speech (e.g. identity, emotion and hesitation). It is also very robust to background noise and can handle audio recorded in a variety of situations (e.g. multiple people speaking). Recent approaches employ end-to-end deep learning taking into account both audio and text as input to generate facial animation for the whole face. However, scarcity of publicly available expressive audio-3D facial animation datasets poses a major bottleneck. The res…
PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features
Macau University of Science and Technology（澳门科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology Beijing（北京科技大学）；Mashang Consumer Finance Co., Ltd（马上消费金融股份有限公司）
Zhejiang University（浙江大学）；International Design Institute of Zhejiang University（浙江大学国际设计研究院）
Formulated as a conditional generation problem, face animation aims at synthesizing continuous face images from a single source image driven by a set of conditional face motion. Previous works mainly model the face motion as conditions with 1D or 2D representation (e.g., action units, emotion codes, landmark), which often leads to low-quality results in some complicated scenarios such as continuous generation and largepose transformation. To tackle this problem, the conditions are supposed to meet two requirements, i.e., motion information preserving and geometric continuity. To this end, we p…
3DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Microsoft Mixed Reality & AI Lab（微软混合现实与人工智能实验室）；Technical University of Darmstadt（达姆施塔特工业大学）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Talking face synthesis has been widely studied in either appearance-based or warping-based methods. Previous works mostly utilize single face image as a source, and generate novel facial animations by merging other person's facial features. However, some facial regions like eyes or teeth, which may be hidden in the source image, can not be synthesized faithfully and stably. In this paper, We present a landmark driven two-stream network to generate faithful talking facial animation, in which more facial details are created, preserved and transferred from multiple source images instead of a sing…
AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models
Westlake University（西湖大学）；Zhejiang University（浙江大学）；Tiangong University（天工大学）；Hunan University（湖南大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Controllable Expressive 3D Facial Animation via Diffusion in a Unified Multimodal Space
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Laboratory for Big Data and Decision, School of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院大数据与决策实验室）
3DiFACE: Synthesizing and Editing Holistic 3D Facial Animation
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck ETH Center for Intelligent Systems（马克斯·普朗克-苏黎世联邦理工学院智能系统中心）；Technical University of Darmstadt（德累斯顿技术大学）；ETH Zürich（苏黎世联邦理工学院）；Microsoft Mixed Reality & AI Lab, UK（英国微软混合现实与人工智能实验室）
本文提出 3DiFACE，利用全卷积扩散、说话风格个性化与稀疏引导运动扩散，从单一音频生成并可关键帧编辑多样且真实的整体 3D 面部动画。
DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer
The Hong Kong Polytechnic University（香港理工大学）；Center for Artificial Intelligence and Robotics, HKISI CAS（香港中文大学（深圳）人工智能与机器人中心（香港中文大学（深圳）与中国科学院深圳先进技术研究院共建））；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；OPPO Research（OPPO研究院）；SenseTime Research（商汤科技研究院）
PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS部）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Psyche AI.INC（Psyche AI公司）；HKUST（香港科技大学）；CAIR, HKISI, Chinese Academy of Sciences（中国科学院计算智能研究所）；SCSE, FIE, M.U.S.T（M.U.S.T的SCSE、FIE部门）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Microsoft Mixed Reality & AI Lab（微软混合现实与人工智能实验室）
Speech-driven 3D facial animation has been widely explored, with applications in gaming, character animation, virtual reality, and telepresence systems. State-of-the-art methods deform the face topology of the target actor to sync the input audio without considering the identity-specific speaking style and facial idiosyncrasies of the target actor, thus, resulting in unrealistic and inaccurate lip movements. To address this, we present Imitator, a speech-driven facial expression synthesis method, which learns identity-specific details from a short input video and produces novel facial expressi…
We present a deep learning framework for real-time speech-driven 3D facial animation from just raw waveforms. Our deep neural network directly maps an input sequence of speech audio to a series of micro facial action unit activations and head rotations to drive a 3D blendshape face model. In particular, our deep model is able to learn the latent representations of time-varying contextual information and affective states within the speech. Hence, our model not only activates appropriate facial action units at inference to depict different utterance generating actions, in the form of lip movemen…
École Polytechnique Montréal（蒙特利尔综合理工学院）；Université de Montréal（蒙特利尔大学）
We present techniques for improving performance driven facial animation, emotion recognition, and facial key-point or landmark prediction using learned identity invariant representations. Established approaches to these problems can work well if sufficient examples and labels for a particular identity are available and factors of variation are highly controlled. However, labeled examples of facial expressions, emotions and key-points for new individuals are difficult and costly to obtain. In this paper we improve the ability of techniques to generalize to new and unseen individuals by explicit…
EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models
University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology Beijing（北京科技大学）；The New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室模式识别新实验室）；The New Laboratory of Pattern Recognition (NLPR)（模式识别新实验室）
Nanchang Hangkong University（南昌航空大学）；Dalian University of Technology（大连理工大学）；Cardiff University（卡迪夫大学）
Most of the existing audio-driven 3D facial animation methods suffered from the lack of detailed facial expression and head pose, resulting in unsatisfactory experience of human-robot interaction. In this paper, a novel pose-controllable 3D facial animation synthesis method is proposed by utilizing hierarchical audio-vertex attention. To synthesize real and detailed expression, a hierarchical decomposition strategy is proposed to encode the audio signal into both a global latent feature and a local vertex-wise control feature. Then the local and global audio features combined with vertex spati…
