解锁姿态多样性：用于音频驱动说话人像的准确高效的基于隐式关键点的时空扩散模型
Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portrait
AI总结：
针对音频驱动单图像说话人像生成中身份保持、细节还原、姿态多样性与效率难以兼顾的问题，提出KDTalker，融合无监督隐式3D关键点与时空扩散注意力，实现准确唇形同步和高质量多样化动画。
摘要：
音频驱动的单图像说话人像生成在虚拟现实、数字人创建和电影制作中发挥着关键作用。现有方法通常分为基于关键点和基于图像的方法。基于关键点的方法能有效保留人物身份，但由于3D可变形模型（3D Morphable Model）的固定点限制，难以捕捉精细面部细节。此外，传统生成网络在有限数据集上建立音频与关键点之间的因果关系面临挑战，导致姿态多样性较低。相比之下，基于图像的方法利用扩散网络生成具有多样细节的高质量人像，但会产生身份失真且计算成本高昂。本文提出KDTalker，这是首个将无监督隐式3D关键点与时空扩散模型相结合的框架。借助无监督隐式3D关键点，KDTalker能够适应面部信息密度，使扩散过程可以建模多样的头部姿态并灵活捕捉精细面部细节。定制设计的时空注意力机制确保准确的唇形同步，在提高计算效率的同时生成时间一致的高质量动画。实验结果表明，KDTalker在唇形同步准确性、头部姿态多样性和执行效率方面均达到最先进性能。代码可在https://github.com/chaolongy/KDTalker获取。
英文摘要：
Audio-driven single-image talking portrait generation plays a crucial role in virtual reality, digital human creation, and filmmaking. Existing approaches are generally categorized into keypoint-based and image-based methods. Keypoint-based methods effectively preserve character identity but struggle to capture fine facial details due to the fixed points limitation of the 3D Morphable Model. Moreover, traditional generative networks face challenges in establishing causality between audio and keypoints on limited datasets, resulting in low pose diversity. In contrast, image-based approaches produce high-quality portraits with diverse details using the diffusion network but incur identity distortion and expensive computational costs. In this work, we propose KDTalker, the first framework to combine unsupervised implicit 3D keypoint with a spatiotemporal diffusion model. Leveraging unsupervised implicit 3D keypoints, KDTalker adapts facial information densities, allowing the diffusion process to model diverse head poses and capture fine facial details flexibly. The custom-designed spatiotemporal attention mechanism ensures accurate lip synchronization, producing temporally consistent, high-quality animations while enhancing computational efficiency. Experimental results demonstrate that KDTalker achieves state-of-the-art performance regarding lip synchronization accuracy, head pose diversity, and execution efficiency.Our codes are available at https://github.com/chaolongy/KDTalker.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
