无需训练的视频编辑 via 光流增强的分数蒸馏
Training-Free Video Editing via Optical Flow-Enhanced Score Distillation
AI总结：
本研究提出基于预训练文本到视频模型的分数蒸馏方法，通过迭代优化和光流增强技术，解决无需训练视频编辑中的内容保留与时间连续性问题。
摘要：
视觉生成的快速发展，特别是预训练的文本到图像和文本到视频模型的出现，推动了无需训练的视频编辑研究的快速增长。类似于无需训练的图像编辑技术，当前的方法通过视频输入反向和在推理过程中操纵中间特征和注意力来保留原始视频信息，以实现内容编辑。尽管它们已经展示了有希望的结果，但反向过程的有损性质在保持未编辑区域方面提出了显著的挑战。此外，在推理过程中特征和注意力的操纵可能导致意外的过度编辑，并面临局部时间连续性和全局内容一致性方面的挑战。为了解决这些挑战，本研究提出了一种基于预训练文本到视频模型的分数蒸馏范式，其中原始视频通过多个步骤逐步优化，这些步骤由分数蒸馏提供的编辑梯度引导，最终获得目标视频。从原始视频开始的迭代优化，结合内容保留损失，确保保留原始视频中的未编辑区域并抑制过度编辑。为进一步保证视频内容一致性和时间连续性，我们还引入了全局一致性辅助损失和基于光流预测的局部编辑梯度平滑。实验表明，这些策略有效解决了上述挑战，在多个维度上实现了与最先进方法相当或更优的性能，包括未编辑区域的保留、局部时间连续性和编辑结果的全局内容一致性。
英文摘要：
The rapid advancement in visual generation, particularly the emergence of pre-trained text-to-image and text-to-video models, has catalyzed growing interest in training-free video editing research. Mirroring training-free image editing techniques, current approaches preserve original video information through video input inversion and manipulating intermediate features and attention during the inference process to achieve content editing. Although they have demonstrated promising results, the lossy nature of the inversion process poses significant challenges in maintaining unedited regions of the video. Furthermore, feature and attention manipulation during inference can lead to unintended over-editing and face challenges in both local temporal continuity and global content consistency. To address these challenges, this study proposes a score distillation paradigm based on pre-trained text-to-video models, where the original video is iteratively optimized through multiple steps guided by editing gradients provided by score distillation to ultimately obtain the target video. The iterative optimization starting from the original video, combined with content preservation loss, ensures the maintenance of unedited regions in the original video and suppresses over-editing. To further guarantee video content consistency and temporal continuity, we additionally introduce a global consistency auxiliary loss and optical flow prediction-based local editing gradient smoothing. Experiments demonstrate that these strategies effectively address the aforementioned challenges, achieving comparable or superior performance across multiple dimensions including preservation of unedited regions, local temporal continuity, and global content consistency of editing results, compared to state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
