由多模态生成先验赋能的肖像视频编辑
Portrait Video Editing Empowered by Multimodal Generative Priors
AI总结：
提出PortraitGen肖像视频编辑方法，通过构建动态3D Gaussian场和Neural Gaussian Texture机制，结合多模态生成先验与表情指导，实现了高一致性、超100FPS且支持文本与图像驱动等多种编辑应用的高质量视频编辑。
摘要：
我们引入了PortraitGen，一种强大的肖像视频编辑方法，能够通过多模态提示实现一致且富有表现力的风格化。传统的肖像视频编辑方法通常在3D和时间一致性方面存在困难，并且在渲染质量和效率上通常表现不足。为了解决这些问题，我们将肖像视频帧提升为统一的动态3D Gaussian场，从而确保跨帧的结构和时间连贯性。此外，我们设计了一种新颖的Neural Gaussian Texture机制，不仅支持复杂的风格编辑，还实现了超过100FPS的渲染速度。我们的方法通过从大规模2D生成模型中蒸馏出的知识来融合多模态输入。系统还结合了表情相似性指导和人脸感知肖像编辑模块，有效缓解了与迭代数据集更新相关的退化问题。大量实验证明了我们方法的时间一致性、编辑效率和卓越的渲染质量。通过包括文本驱动编辑、图像驱动编辑和重新打光在内的多种应用，展示了所提方法的广泛适用性，突显了其在推动视频编辑领域发展方面的巨大潜力。演示视频和已发布的代码可在我们的项目页面获取：https://ustc3dv.github.io/PortraitGen/
英文摘要：
We introduce PortraitGen, a powerful portrait video editing method that achieves consistent and expressive stylization with multimodal prompts. Traditional portrait video editing methods often struggle with 3D and temporal consistency, and typically lack in rendering quality and efficiency. To address these issues, we lift the portrait video frames to a unified dynamic 3D Gaussian field, which ensures structural and temporal coherence across frames. Furthermore, we design a novel Neural Gaussian Texture mechanism that not only enables sophisticated style editing but also achieves rendering speed over 100FPS. Our approach incorporates multimodal inputs through knowledge distilled from large-scale 2D generative models. Our system also incorporates expression similarity guidance and a face-aware portrait editing module, effectively mitigating degradation issues associated with iterative dataset updates. Extensive experiments demonstrate the temporal consistency, editing efficiency, and superior rendering quality of our method. The broad applicability of the proposed approach is demonstrated through various applications, including text-driven editing, image-driven editing, and relighting, highlighting its great potential to advance the field of video editing. Demo videos and released code are provided in our project page: https://ustc3dv.github.io/PortraitGen/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
