arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.07385cs.CV
收藏

GSmoothFace:通过细粒度3D人脸引导实现通用的流畅说话人脸生成

GSmoothFace: Generalized Smooth Talking Face Generation via Fine Grained 3D Face Guidance

The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));The Future Network of Intelligence Institute(未来智能网络研究院);School of Science and Engineering(理工学院);Sun Yat-sen University(中山大学);Shenzhen University(深圳大学);Health Science Center(健康科学中心);South China Hospital(华南医院)

Haiming Zhang, Zhihao Yuan, Chaoda Zheng, Xu Yan, Baoyuan Wang, Guanbin Li, Song Wu, Shuguang Cui, Zhen Li

AI总结:

针对现有语音驱动说话人脸生成需特定形象训练、唇部运动不稳定的问题,提出由细粒度3D人脸引导的两阶段GSmoothFace模型,通过A2EP和TAFT模块实现高真实感、强泛化性的流畅说话人脸生成。

摘要:

尽管现有的语音驱动说话人脸生成方法已取得显著进展,但由于需要针对特定虚拟形象训练且唇部运动不稳定,它们距离实际应用仍有较大差距。为解决上述问题,我们提出了GSmoothFace,这是一种由细粒度3D人脸模型引导的新型两阶段通用说话人脸生成模型,能够在保留说话人身份的同时合成流畅的唇部动态效果。\n我们提出的GSmoothFace模型主要由音频到表情预测(Audio to Expression Prediction, A2EP)模块和目标自适应人脸翻译(Target Adaptive Face Translation, TAFT)模块组成。具体而言,我们首先开发了A2EP模块,用于预测与驱动语音同步的表情参数。该模块使用transformer捕捉长程音频上下文,并从细粒度3D人脸顶点中学习参数,从而实现准确且流畅的唇形同步性能。\n随后,经过精心设计的TAFT模块在形态增强人脸融合(Morphology Augmented Face Blending, MAFB)的支持下,以预测得到的表情参数和目标视频为输入,在不扭曲背景内容的前提下修改目标视频的人脸区域。TAFT模块有效利用了目标视频中的身份外观和背景上下文,无需重新训练即可泛化到不同说话人。定量和定性实验均证实了我们的方法在真实感、唇形同步性和视觉质量方面的优越性。代码、数据及预训练模型申请可查看项目页面:https://zhanghm1995.github.io/GSmoothFace。

英文摘要:

Although existing speech-driven talking face generation methods achieve significant progress, they are far from real-world application due to the avatar-specific training demand and unstable lip movements. To address the above issues, we propose the GSmoothFace, a novel two-stage generalized talking face generation model guided by a fine-grained 3d face model, which can synthesize smooth lip dynamics while preserving the speaker's identity. Our proposed GSmoothFace model mainly consists of the Audio to Expression Prediction (A2EP) module and the Target Adaptive Face Translation (TAFT) module. Specifically, we first develop the A2EP module to predict expression parameters synchronized with the driven speech. It uses a transformer to capture the long-term audio context and learns the parameters from the fine-grained 3D facial vertices, resulting in accurate and smooth lip-synchronization performance. Afterward, the well-designed TAFT module, empowered by Morphology Augmented Face Blending (MAFB), takes the predicted expression parameters and target video as inputs to modify the facial region of the target video without distorting the background content. The TAFT effectively exploits the identity appearance and background context in the target video, which makes it possible to generalize to different speakers without retraining. Both quantitative and qualitative experiments confirm the superiority of our method in terms of realism, lip synchronization, and visual quality. See the project page for code, data, and request pre-trained models: https://zhanghm1995.github.io/GSmoothFace.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑