PerformRecast：基于驱动视频的表情与头部姿态解耦的肖像视频编辑
PerformRecast: Expression and Head Pose Disentanglement for Portrait Video Editing
AI总结：
本文提出PerformRecast，通过改进关键点变换公式和分离面部与非面部区域，实现表情与头部姿态的解耦，提升肖像视频编辑的可控性和效率。
摘要：
本文主要研究基于驱动视频的表情-only肖像视频性能编辑任务，对动画和电影行业具有重要作用。现有研究主要关注肖像动画，旨在根据驱动视频中的面部运动来动画化静态肖像图像。然而，它们难以将面部表情与头部姿态旋转解耦，因此缺乏独立编辑面部表情的能力。本文提出PerformRecast，一种多功能的表情-only视频编辑方法，专门用于重制现有电影和动画中的表演。我们的方法关键见解来源于3D可形变面部模型（3DMM）的特性，该模型通过独立参数对3D面部网格的身份、面部表情和头部姿态进行建模。因此，我们改进了先前方法中的关键点变换公式，使其更符合3DMM模型，从而实现更好的解耦并为用户提供更精细的控制。此外，为了防止生成结果中面部边界处的对齐错误，我们分离输入肖像图像的面部和非面部区域，并预训练一个教师模型，为它们提供单独的监督。大量实验表明，我们的方法产生了高质量的结果，更忠实于驱动视频，在可控性和效率方面均优于现有方法。我们的代码、数据和训练模型可在https://youku-aigc.github.io/PerformRecast获取。
英文摘要：
This paper primarily investigates the task of expression-only portrait video performance editing based on a driving video, which plays a crucial role in animation and film industries. Most existing research mainly focuses on portrait animation, which aims to animate a static portrait image according to the facial motion from the driving video. As a consequence, it remains challenging for them to disentangle the facial expression from head pose rotation and thus lack the ability to edit facial expression independently. In this paper, we propose PerformRecast, a versatile expression-only video editing method which is dedicated to recast the performance in existing film and animation. The key insight of our method comes from the characteristics of 3D Morphable Face Model (3DMM), which models the face identity, facial expression and head pose of 3D face mesh with separate parameters. Therefore, we improve the keypoints transformation formula in previous methods to make it more consistent with 3DMM model, which achieves a better disentanglement and provides users with much more fine-grained control. Furthermore, to avoid the misalignment around the boundary of face in generated results, we decouple the facial and non-facial regions of input portrait images and pre-train a teacher model to provide separate supervision for them. Extensive experiments show that our method produces high-quality results which are more faithful to the driving video, outperforming existing methods in both controllability and efficiency. Our code, data and trained models are available at https://youku-aigc.github.io/PerformRecast.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
