FactorPortrait: 通过解耦的表情、姿态和视角实现可控的肖像动画
FactorPortrait: Controllable Portrait Animation via Disentangled Expression, Pose, and Viewpoint
AI总结：
FactorPortrait通过解耦表情、姿态和视角实现可控肖像动画，利用预训练编码器和3D网格跟踪生成逼真动态效果。
摘要：
我们介绍了FactorPortrait，一种用于可控肖像动画的视频扩散方法，能够从解耦的表情、头部运动和摄像机视角的控制信号中生成逼真的合成效果。给定一张肖像图像、驱动视频和摄像机轨迹，我们的方法通过将驱动视频中的面部表情和头部运动转移到肖像上，同时允许从任意视角合成新的视角。我们利用预训练的图像编码器从驱动视频中提取面部表情潜在特征作为动画生成的控制信号。这些潜在特征隐式地捕捉了细腻的面部表情动态，包含了身份和姿态信息的解耦，并通过我们提出的表情控制器高效地注入到视频扩散变换器中。对于摄像机和头部姿态控制，我们采用Plücker射线图和从3D身体网格跟踪中渲染的法线图。为了训练我们的模型，我们整理了一个包含多样摄像机视角、头部姿态和面部表情动态组合的大型合成数据集。广泛的实验表明，我们的方法在真实感、表达性、控制精度和视角一致性方面均优于现有方法。
英文摘要：
We introduce FactorPortrait, a video diffusion method for controllable portrait animation that enables lifelike synthesis from disentangled control signals of facial expressions, head movement, and camera viewpoints. Given a single portrait image, a driving video, and camera trajectories, our method animates the portrait by transferring facial expressions and head movements from the driving video while simultaneously enabling novel view synthesis from arbitrary viewpoints. We utilize a pre-trained image encoder to extract facial expression latents from the driving video as control signals for animation generation. Such latents implicitly capture nuanced facial expression dynamics with identity and pose information disentangled, and they are efficiently injected into the video diffusion transformer through our proposed expression controller. For camera and head pose control, we employ Plücker ray maps and normal maps rendered from 3D body mesh tracking. To train our model, we curate a large-scale synthetic dataset containing diverse combinations of camera viewpoints, head poses, and facial expression dynamics. Extensive experiments demonstrate that our method outperforms existing approaches in realism, expressiveness, control accuracy, and view consistency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
