DynVideo-E：利用动态 NeRF 进行大规模运动与视角变化的以人为中心的视频编辑
DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing
AI总结：
本文提出 DynVideo-E，利用动态 NeRF 与多视角多姿态 SDS 等设计实现大规模运动及视角变化下的以人为中心视频编辑，并在人类偏好上大幅超越现有方法。
摘要：
尽管基于扩散模型的视频编辑近期取得了进展，但由于长程一致性与逐帧编辑之间的矛盾，现有方法仍局限于短视频。以往通过引入 video-2D 表征来解决这一挑战的尝试，在处理大规模运动和视角变化的视频时遇到显著困难，尤其是在以人为中心的场景中。为克服该问题，我们提出引入动态 Neural Radiance Fields（NeRF，神经辐射场）作为创新的视频表征，使编辑能够在 3D 空间中执行，并通过形变场传播到整个视频。为提供一致且可控的编辑，我们提出基于图像的 video-NeRF 编辑流程，并包含一系列创新设计，包括来自 2D 个性化扩散先验和 3D 扩散先验的多视角多姿态 Score Distillation Sampling（SDS，分数蒸馏采样）、重建损失、文本引导的局部部件超分辨率以及风格迁移。大量实验表明，我们名为 DynVideo-E 的方法在两个具有挑战性的数据集上显著优于 SOTA 方法，在人类偏好上以 50%～95% 的大幅幅度领先。代码将发布于 https://showlab.github.io/DynVideo-E/。
英文摘要：
Despite recent progress in diffusion-based video editing, existing methods are limited to short-length videos due to the contradiction between long-range consistency and frame-wise editing. Prior attempts to address this challenge by introducing video-2D representations encounter significant difficulties with large-scale motion- and view-change videos, especially in human-centric scenarios. To overcome this, we propose to introduce the dynamic Neural Radiance Fields (NeRF) as the innovative video representation, where the editing can be performed in the 3D spaces and propagated to the entire video via the deformation field. To provide consistent and controllable editing, we propose the image-based video-NeRF editing pipeline with a set of innovative designs, including multi-view multi-pose Score Distillation Sampling (SDS) from both the 2D personalized diffusion prior and 3D diffusion prior, reconstruction losses, text-guided local parts super-resolution, and style transfer. Extensive experiments demonstrate that our method, dubbed as DynVideo-E, significantly outperforms SOTA approaches on two challenging datasets by a large margin of 50% ~ 95% for human preference. Code will be released at https://showlab.github.io/DynVideo-E/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
