arXiv 2506.147422025-06-17cs.CV

SyncTalk++：基于高斯泼溅的高保真高效同步说话人头部合成

SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splatting

School of Information, Renmin University of China（中国人民大学信息学院）；School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（北京邮电大学信息与通信工程学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；School of Economics and Management, Tsinghua University（清华大学经济管理学院）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）

Ziqiao Peng, Wentao Hu, Junyuan Ma, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Hui Tian, Jun He, Hongyan Liu, Zhaoxin Fan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对语音驱动说话人头部合成的同步性难题，提出SyncTalk++框架，通过高斯泼溅渲染、多模块同步控制及鲁棒性增强，实现了高真实感、高同步性且速度达101帧/秒的合成效果，性能优于现有最优方法。

摘要：

在合成逼真的语音驱动说话人头部视频时实现高度同步是一项重大挑战。逼真的说话人头部需要主体身份、唇部动作、面部表情和头部姿态的协同一致，缺乏这些同步性是核心缺陷，会导致结果失真。为解决被称为打造逼真说话人头部过程中“恶魔”的同步关键问题，我们提出SyncTalk++：它配备基于高斯泼溅的动态肖像渲染器，确保稳定保留主体身份；还设有面部同步控制器，使唇部动作与语音对齐，同时创新性地使用3D面部blendshape模型重建精准的面部表情。为实现自然的头部动作，我们提出头部同步稳定器，优化头部姿态以提升稳定性。此外，SyncTalk++通过加入表情生成器和躯干修复器，分别生成与语音匹配的面部表情和无缝的躯干区域，增强了对分布外（OOD）音频的鲁棒性。我们的方法保持了帧间视觉细节的一致性与连续性，显著提升了渲染速度和质量，最高可达101帧每秒。大量实验和用户研究表明，SyncTalk++在同步性和真实感上优于现有最优方法。建议观看补充视频：https://ziqiaopeng.github.io/synctalk++

英文摘要：

Achieving high synchronization in the synthesis of realistic, speech-driven talking head videos presents a significant challenge. A lifelike talking head requires synchronized coordination of subject identity, lip movements, facial expressions, and head poses. The absence of these synchronizations is a fundamental flaw, leading to unrealistic results. To address the critical issue of synchronization, identified as the ''devil'' in creating realistic talking heads, we introduce SyncTalk++, which features a Dynamic Portrait Renderer with Gaussian Splatting to ensure consistent subject identity preservation and a Face-Sync Controller that aligns lip movements with speech while innovatively using a 3D facial blendshape model to reconstruct accurate facial expressions. To ensure natural head movements, we propose a Head-Sync Stabilizer, which optimizes head poses for greater stability. Additionally, SyncTalk++ enhances robustness to out-of-distribution (OOD) audio by incorporating an Expression Generator and a Torso Restorer, which generate speech-matched facial expressions and seamless torso regions. Our approach maintains consistency and continuity in visual details across frames and significantly improves rendering speed and quality, achieving up to 101 frames per second. Extensive experiments and user studies demonstrate that SyncTalk++ outperforms state-of-the-art methods in synchronization and realism. We recommend watching the supplementary video: https://ziqiaopeng.github.io/synctalk++.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。