可控说话人脸生成：基于隐式面部关键点编辑
Controllable Talking Face Generation by Implicit Facial Keypoints Editing
AI总结：
提出ControlTalk，一种基于隐式关键点编辑的音频驱动说话人脸生成方法，统一处理图像和视频输入，实现精确唇同步和嘴形控制，在HDTF和MEAD基准上优于现有方法。
摘要：
音频驱动的说话人脸生成在数字人研究领域引起了广泛关注。现有方法受限于复杂的模型架构，这些架构相互依赖，使得对图像或视频输入的重新编辑过程变得复杂。在本工作中，我们提出了ControlTalk，一种基于驱动音频控制面部表情变形的说话人脸生成方法，该方法能够以统一的方式为单张图像或连续视频输入构建头部姿态和面部表情（包括嘴唇运动）。通过利用预训练的视频合成渲染器并提出轻量级适配，ControlTalk实现了精确且自然的唇部同步，同时能够对嘴部张开形状进行定量控制。我们的实验表明，在广泛使用的基准（包括HDTF和MEAD）上，我们的方法优于最先进的性能。参数化适配展示了卓越的泛化能力，能够有效处理同身份和跨身份场景下的表情变形，并将其效用扩展到域外人像，无论语言如何。代码可在https://github.com/NetEase-Media/ControlTalk获取。
英文摘要：
Audio-driven talking face generation has garnered significant interest within the domain of digital human research. Existing methods are encumbered by intricate model architectures that are intricately dependent on each other, complicating the process of re-editing image or video inputs. In this work, we present ControlTalk, a talking face generation method to control face expression deformation based on driven audio, which can construct the head pose and facial expression including lip motion for both single image or sequential video inputs in a unified manner. By utilizing a pre-trained video synthesis renderer and proposing the lightweight adaptation, ControlTalk achieves precise and naturalistic lip synchronization while enabling quantitative control over mouth opening shape. Our experiments show that our method is superior to state-of-the-art performance on widely used benchmarks, including HDTF and MEAD. The parameterized adaptation demonstrates remarkable generalization capabilities, effectively handling expression deformation across same-ID and cross-ID scenarios, and extending its utility to out-of-domain portraits, regardless of languages. Code is available at https://github.com/NetEase-Media/ControlTalk.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
