JoyGen：音频驱动的3D深度感知说话人脸视频编辑
JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing
AI总结：
JoyGen提出两阶段框架，结合3D重建、audio2motion和深度图监督实现音频驱动的精确唇音同步说话人脸编辑，并构建130小时中文数据集验证其优越性能。
摘要：
说话人脸视频生成研究已取得显著进展；然而，在基于输入音频编辑唇形时，精确的唇音同步和高视觉质量仍具挑战性。本文提出JoyGen，一个新颖的两阶段说话人脸生成框架，包括音频驱动的唇部运动生成和视觉外观合成。在第一阶段，3D重建模型和audio2motion模型分别预测身份系数和表情系数。接着，通过将音频特征与人脸深度图相结合，我们为人脸生成中的精确唇音同步提供全面监督。此外，我们构建了一个包含130小时高质量视频的中文说话人脸数据集。JoyGen在开源的HDTF数据集和我们整理的数据集上进行训练。实验结果表明，我们的方法实现了优越的唇音同步和视觉质量。
英文摘要：
Significant progress has been made in talking-face video generation research; however, precise lip-audio synchronization and high visual quality remain challenging in editing lip shapes based on input audio. This paper introduces JoyGen, a novel two-stage framework for talking-face generation, comprising audio-driven lip motion generation and visual appearance synthesis. In the first stage, a 3D reconstruction model and an audio2motion model predict identity and expression coefficients respectively. Next, by integrating audio features with a facial depth map, we provide comprehensive supervision for precise lip-audio synchronization in facial generation. Additionally, we constructed a Chinese talking-face dataset containing 130 hours of high-quality video. JoyGen is trained on the open-source HDTF dataset and our curated dataset. Experimental results demonstrate superior lip-audio synchronization and visual quality achieved by our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
