基于GAN的真实感地标引导人脸视频修复研究
Towards Realistic Landmark-Guided Facial Video Inpainting Based on GANs
AI总结：
针对人脸视频遮挡去除需求，提出基于GAN的表情引导修复网络，结合人脸关键点、无遮挡参考图与定制FER损失，实现静态/动态遮挡的自适应去除，保障帧间身份一致与时空连贯的逼真修复效果。
摘要：
人脸视频修复在众多应用中发挥着关键作用，包括但不限于视频会议与远程医疗中的遮挡去除、人脸表情分析增强、隐私保护、图形叠加层整合以及虚拟化妆。由于面部特征的复杂性以及人类天生对人脸的熟悉度，该领域面临着严峻挑战，这也对修复结果的准确性和逼真度提出了更高要求。针对该场景下的遮挡去除难题，本研究聚焦于从被掩码覆盖的人脸数据中生成完整图像的渐进式任务，同时确保空间和时间一致性。本研究提出了一种面向表情的视频修复网络，采用生成对抗网络（GANs）处理所有帧中的静态和动态遮挡。通过利用人脸关键点（facial landmarks）和无遮挡参考图像，模型可在帧间保持用户身份的一致性。我们还通过定制化的面部表情识别（facial expression recognition, FER）损失函数进一步增强情感保留效果，确保修复输出细节丰富。我们提出的框架能够自适应地去除人脸视频中的遮挡——无论遮挡在帧中是静态还是动态的，同时生成逼真且连贯的结果。
英文摘要：
Facial video inpainting plays a crucial role in a wide range of applications, including but not limited to the removal of obstructions in video conferencing and telemedicine, enhancement of facial expression analysis, privacy protection, integration of graphical overlays, and virtual makeup. This domain presents serious challenges due to the intricate nature of facial features and the inherent human familiarity with faces, heightening the need for accurate and persuasive completions. In addressing challenges specifically related to occlusion removal in this context, our focus is on the progressive task of generating complete images from facial data covered by masks, ensuring both spatial and temporal coherence. Our study introduces a network designed for expression-based video inpainting, employing generative adversarial networks (GANs) to handle static and moving occlusions across all frames. By utilizing facial landmarks and an occlusion-free reference image, our model maintains the user's identity consistently across frames. We further enhance emotional preservation through a customized facial expression recognition (FER) loss function, ensuring detailed inpainted outputs. Our proposed framework exhibits proficiency in eliminating occlusions from facial videos in an adaptive form, whether appearing static or dynamic on the frames, while providing realistic and coherent results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
