用于说话人头部视频生成的隐式身份表示条件化记忆补偿网络
Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head video Generation
AI总结：
针对复杂运动导致说话人头部视频生成出现遮挡伪影和细节退化的问题，本文提出 MCNet，利用全局面部元记忆库与隐式身份查询补偿源特征，并在 VoxCeleb1 和 CelebV 上取得优于现有方法的结果。
摘要：
说话人头部视频生成旨在利用从目标驱动视频中获得的运动信息，使静态图像中的人脸呈现动态姿态和表情，同时保持源图像中人物的身份。然而，驱动视频中剧烈且复杂的运动会导致生成结果模糊，因为静态源图像无法为遮挡区域或细微的表情变化提供足够的外观信息，这会产生严重伪影并显著降低生成质量。为解决该问题，我们提出学习一个全局面部表示空间，并设计了一种新颖的隐式身份表示条件化记忆补偿网络，命名为 MCNet，用于高保真说话人头部生成。具体而言，我们设计了一个网络模块，从所有训练样本中学习统一的空间面部元记忆库，该记忆库能够提供丰富的面部结构和外观先验，以补偿生成过程中经形变处理的源面部特征。此外，我们基于从源图像的离散关键点学习到的隐式身份表示，提出了一种有效的查询机制。该机制能够极大地促进从记忆库中检索更相关的信息以进行补偿。大量实验表明，MCNet 可以学习具有代表性和互补性的面部记忆，并且在 VoxCeleb1 和 CelebV 数据集上明显优于以往最先进的说话人头部生成方法。请查看我们的项目。
英文摘要：
Talking head video generation aims to animate a human face in a still image with dynamic poses and expressions using motion information derived from a target-driving video, while maintaining the person's identity in the source image. However, dramatic and complex motions in the driving video cause ambiguous generation, because the still source image cannot provide sufficient appearance information for occluded regions or delicate expression variations, which produces severe artifacts and significantly degrades the generation quality. To tackle this problem, we propose to learn a global facial representation space, and design a novel implicit identity representation conditioned memory compensation network, coined as MCNet, for high-fidelity talking head generation.~Specifically, we devise a network module to learn a unified spatial facial meta-memory bank from all training samples, which can provide rich facial structure and appearance priors to compensate warped source facial features for the generation. Furthermore, we propose an effective query mechanism based on implicit identity representations learned from the discrete keypoints of the source image. It can greatly facilitate the retrieval of more correlated information from the memory bank for the compensation. Extensive experiments demonstrate that MCNet can learn representative and complementary facial memory, and can clearly outperform previous state-of-the-art talking head generation methods on VoxCeleb1 and CelebV datasets. Please check our \href{https://github.com/harlanhong/ICCV2023-MCNET}{Project}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
