CP-EB：具有可控姿态和眨眼嵌入的说话人脸生成
CP-EB: Talking Face Generation with Controllable Pose and Eye Blinking Embedding
AI总结：
提出CP-EB方法，基于GAN从音频和参考视频提取眨眼特征并对比训练，嵌入身份与姿态特征，生成同步嘴唇、自然头部姿态和眨眼的逼真说话人脸。
摘要：
本文提出了一种名为“CP-EB”的说话人脸生成方法，该方法以音频信号为输入，以人物图像为参考，合成逼真的人物说话视频，其中头部姿态由短视频片段控制，并嵌入适当的眨眼动作。值得注意的是，头部姿态和眨眼对于深度伪造检测都是重要的方面。通过视频对姿态的隐式控制已在最先进的工作中实现。根据最近的研究，眨眼与输入音频的相关性较弱，这意味着从音频中提取眨眼特征并生成眨眼动作是可能的。因此，我们提出了一种基于GAN的架构，分别从输入音频和参考视频中提取眨眼特征，并在它们之间进行对比训练，然后将其嵌入身份和姿态的拼接特征中，以生成说话人脸图像。实验结果表明，所提出的方法能够生成具有同步嘴唇动作、自然头部姿态和眨眼动作的逼真说话人脸。
英文摘要：
This paper proposes a talking face generation method named "CP-EB" that takes an audio signal as input and a person image as reference, to synthesize a photo-realistic people talking video with head poses controlled by a short video clip and proper eye blinking embedding. It's noted that not only the head pose but also eye blinking are both important aspects for deep fake detection. The implicit control of poses by video has already achieved by the state-of-art work. According to recent research, eye blinking has weak correlation with input audio which means eye blinks extraction from audio and generation are possible. Hence, we propose a GAN-based architecture to extract eye blink feature from input audio and reference video respectively and employ contrastive training between them, then embed it into the concatenated features of identity and poses to generate talking face images. Experimental results show that the proposed method can generate photo-realistic talking face with synchronous lips motions, natural head poses and blinking eyes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
