MegActor: 利用原始视频的力量实现生动的人像动画
MegActor: Harness the Power of Raw Video for Vivid Portrait Animation
AI总结：
针对原始视频驱动人像动画存在的身份泄漏、背景与面部细节干扰问题，提出MegActor条件扩散模型，通过合成数据、背景编码整合、风格迁移实现效果提升，仅用公开数据集训练即达商业模型水平。
摘要：
在人像动画领域，尽管原始驱动视频比 landmarks（关键点）等中间表征包含更丰富的面部表情信息，但它们很少成为研究对象。这是由原始视频驱动的人像动画所固有的两大挑战导致的：1）严重的身份泄漏；2）无关的背景以及皱纹等面部细节会降低性能。为了利用原始视频的力量实现生动的人像动画，我们提出了一种名为MegActor的开创性条件扩散模型。首先，我们引入了一个合成数据生成框架，用于创建动作和表情一致但身份不一致的视频，以缓解身份泄漏问题。其次，我们对参考图像的前景和背景进行分割，并采用CLIP对背景细节进行编码。这些编码后的信息随后通过文本嵌入模块整合到网络中，从而确保背景的稳定性。最后，我们进一步将参考图像的外观风格迁移到驱动视频上，以消除驱动视频中面部细节的影响。我们的最终模型仅在公开数据集上训练，取得了与商业模型相当的结果。我们希望这能为开源社区提供帮助。代码可在https://github.com/megvii-research/MegFaceAnimate 获取。
英文摘要：
Despite raw driving videos contain richer information on facial expressions than intermediate representations such as landmarks in the field of portrait animation, they are seldom the subject of research. This is due to two challenges inherent in portrait animation driven with raw videos: 1) significant identity leakage; 2) Irrelevant background and facial details such as wrinkles degrade performance. To harnesses the power of the raw videos for vivid portrait animation, we proposed a pioneering conditional diffusion model named as MegActor. First, we introduced a synthetic data generation framework for creating videos with consistent motion and expressions but inconsistent IDs to mitigate the issue of ID leakage. Second, we segmented the foreground and background of the reference image and employed CLIP to encode the background details. This encoded information is then integrated into the network via a text embedding module, thereby ensuring the stability of the background. Finally, we further style transfer the appearance of the reference image to the driving video to eliminate the influence of facial details in the driving videos. Our final model was trained solely on public datasets, achieving results comparable to commercial models. We hope this will help the open-source community.The code is available at https://github.com/megvii-research/MegFaceAnimate.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
