GSmoothFace: Generalized Smooth Talking Face Generation via Fine Grained 3D Face Guidance
The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));The Future Network of Intelligence Institute(未来智能网络研究院);School of Science and Engineering(理工学院);Sun Yat-sen University(中山大学);Shenzhen University(深圳大学);Health Science Center(健康科学中心);South China Hospital(华南医院)
Haiming Zhang, Zhihao Yuan, Chaoda Zheng, Xu Yan, Baoyuan Wang, Guanbin Li, Song Wu, Shuguang Cui, Zhen Li
尽管现有的语音驱动说话人脸生成方法已取得显著进展,但由于需要针对特定虚拟形象训练且唇部运动不稳定,它们距离实际应用仍有较大差距。为解决上述问题,我们提出了GSmoothFace,这是一种由细粒度3D人脸模型引导的新型两阶段通用说话人脸生成模型,能够在保留说话人身份的同时合成流畅的唇部动态效果。\n我们提出的GSmoothFace模型主要由音频到表情预测(Audio to Expression Prediction, A2EP)模块和目标自适应人脸翻译(Target Adaptive Face Translation, TAFT)模块组成。具体而言,我们首先开发了A2EP模块,用于预测与驱动语音同步的表情参数。该模块使用transformer捕捉长程音频上下文,并从细粒度3D人脸顶点中学习参数,从而实现准确且流畅的唇形同步性能。\n随后,经过精心设计的TAFT模块在形态增强人脸融合(Morphology Augmented Face Blending, MAFB)的支持下,以预测得到的表情参数和目标视频为输入,在不扭曲背景内容的前提下修改目标视频的人脸区域。TAFT模块有效利用了目标视频中的身份外观和背景上下文,无需重新训练即可泛化到不同说话人。定量和定性实验均证实了我们的方法在真实感、唇形同步性和视觉质量方面的优越性。代码、数据及预训练模型申请可查看项目页面:https://zhanghm1995.github.io/GSmoothFace。
英文摘要:
Although existing speech-driven talking face generation methods achieve significant progress, they are far from real-world application due to the avatar-specific training demand and unstable lip movements. To address the above issues, we propose the GSmoothFace, a novel two-stage generalized talking face generation model guided by a fine-grained 3d face model, which can synthesize smooth lip dynamics while preserving the speaker's identity. Our proposed GSmoothFace model mainly consists of the Audio to Expression Prediction (A2EP) module and the Target Adaptive Face Translation (TAFT) module. Specifically, we first develop the A2EP module to predict expression parameters synchronized with the driven speech. It uses a transformer to capture the long-term audio context and learns the parameters from the fine-grained 3D facial vertices, resulting in accurate and smooth lip-synchronization performance. Afterward, the well-designed TAFT module, empowered by Morphology Augmented Face Blending (MAFB), takes the predicted expression parameters and target video as inputs to modify the facial region of the target video without distorting the background content. The TAFT effectively exploits the identity appearance and background context in the target video, which makes it possible to generalize to different speakers without retraining. Both quantitative and qualitative experiments confirm the superiority of our method in terms of realism, lip synchronization, and visual quality. See the project page for code, data, and request pre-trained models: https://zhanghm1995.github.io/GSmoothFace.
See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Speech and Acoustic Laboratory, Joy Future Academy, Jingdong Corporation(语音与声学实验室、Joy Future Academy、京东公司);Aarhus University(阿arhus大学)
Multimodal Diffusion Transformer with Memory Bank for Scalable Long-Duration Talking Video Generation
South China University of Technology(南方科技大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing National Research Center for Information Science and Technology, Tsinghua University(北京信息科学研究中心,清华大学);Department of Automation, BNRist, Tsinghua University(清华大学自动化系,北京信息科学研究中心)
Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation
School of Software Technology, Zhejiang University(浙江大学软件学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Ningbo Global Innovation Center, Zhejiang University(浙江大学宁波全球创新中心)
AUHead: Realistic Emotional Talking Head Generation via Action Units Control
University of the Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);State Key Laboratory of Communication Content Cognition, People’s Daily Online(人民日報網通信內容認知重點實驗室)
Versatile Multimodal Controls for Expressive Talking Human Animation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi'an Jiaotong University, Ant Group(人机混合增强智能国家级实验室,西安交通大学,蚂蚁集团);National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi'an Jiaotong University(人机混合增强智能国家级实验室,西安交通大学);University at Buffalo(布法罗大学)
Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert
Grad. School of Artificial Intelligence(人工智能研究生院);POSTECH(浦项科技大学);ENSC, Bordeaux INP, France(法国波尔多国立高等工程学院);KRAFTON, Korea(韩国KRAFTON公司);Inst. for Convergence Research and Education in Advanced Technology, Yonsei University, Korea(延世大学先进技术融合研究与教育学院)
EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidance
National University of Singapore(新加坡国立大学);Southern University of Science and Technology(南方科技大学);Fudan University(复旦大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models
University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology Beijing(北京科技大学);The New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统国家重点实验室模式识别新实验室);The New Laboratory of Pattern Recognition (NLPR)(模式识别新实验室)