PropFly: 通过预训练视频扩散模型的实时监督学习进行传播
PropFly: Learning to Propagate via On-the-Fly Supervision from Pre-trained Video Diffusion Models
AI总结:
PropFly通过预训练视频扩散模型的实时监督学习,实现基于传播的视频编辑,无需配对数据集,提升编辑质量和一致性。
摘要:
基于传播的视频编辑通过将单个编辑帧传播到后续帧,同时保持运动和结构等原始上下文,实现了精确的用户控制。然而,训练此类模型需要大规模配对(源和编辑)视频数据集,这些数据集成本高且难以获取。因此,我们提出了PropFly,一种用于基于传播的视频编辑的训练流程,依赖于预训练视频扩散模型(VDMs)的实时监督,而不是需要现成或预计算的配对视频编辑数据集。具体而言,我们的PropFly利用中间噪声潜码的一步干净潜码估计,通过变化的分类器自由引导(CFG)尺度,实时合成多样化的'源'(低CFG)和'编辑'(高CFG)潜码对。源潜码作为视频的结构信息,而编辑潜码提供学习传播的目标转换。我们的流程使预训练VDM上附加一个适配器,通过引导调节流匹配(GMFM)损失学习传播编辑,该损失指导模型复制目标转换。我们的实时监督确保模型学习到时间一致且动态的转换。大量实验表明,我们的PropFly在各种视频编辑任务上显著优于最先进的方法,产生高质量的编辑结果。
英文摘要:
Propagation-based video editing enables precise user control by propagating a single edited frame into following frames while maintaining the original context such as motion and structures. However, training such models requires large-scale, paired (source and edited) video datasets, which are costly and complex to acquire. Hence, we propose the PropFly, a training pipeline for Propagation-based video editing, relying on on-the-Fly supervision from pre-trained video diffusion models (VDMs) instead of requiring off-the-shelf or precomputed paired video editing datasets. Specifically, our PropFly leverages one-step clean latent estimations from intermediate noised latents with varying Classifier-Free Guidance (CFG) scales to synthesize diverse pairs of 'source' (low-CFG) and 'edited' (high-CFG) latents on-the-fly. The source latent serves as structural information of the video, while the edited latent provides the target transformation for learning propagation. Our pipeline enables an additional adapter attached to the pre-trained VDM to learn to propagate edits via Guidance-Modulated Flow Matching (GMFM) loss, which guides the model to replicate the target transformation. Our on-the-fly supervision ensures the model to learn temporally consistent and dynamic transformations. Extensive experiments demonstrate that our PropFly significantly outperforms the state-of-the-art methods on various video editing tasks, producing high-quality editing results.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。