FREE-Edit：使用编辑感知注入在修正流模型中进行零样本图像驱动视频编辑
FREE-Edit: Using Editing-aware Injection in Rectified Flow Models for Zero-shot Image-Driven Video Editing
AI总结：
本文提出FREE-Edit框架，通过编辑感知注入提升图像驱动视频编辑效果，无需微调即可在多种场景中生成高质量输出。
摘要：
图像驱动视频编辑旨在将修改后的第一帧内容传播到其余帧。现有方法通常使用预训练的图像到视频（I2V）模型将源视频转换为噪声，然后通过编辑后的第一帧引导采样过程。通常，保持源视频运动和布局的常用方法是在去噪过程中通过重建时注入注意力。然而，这种注入常导致不理想的结果，过度注入会导致与源视频语义冲突，不足注入则带来有限的源表示。为此，我们提出编辑感知（REE）注入方法，以调节每个token的注入强度。具体来说，我们首先计算源视频与编辑后第一帧的像素差异，形成对应的编辑掩码。然后，通过使用光流跟踪整个视频中的编辑区域，通过变形第一帧掩码。接着，根据生成的编辑感知特征注入强度，对每个token进行相应处理，其中在编辑区域不进行注入。基于REE注入，我们进一步提出一个基于近期兴起的修正流模型的零样本图像驱动视频编辑框架，称为FREE-Edit。无需微调或训练，我们的FREE-Edit在多种图像驱动视频编辑场景中表现出色，展示了其生成比现有技术更高质量输出的能力。项目页面：https://free-edit.github.io/page/.
英文摘要：
Image-driven video editing aims to propagate edit contents from the modified first frame to the remaining frames. Existing methods usually invert the source video to noise using a pre-trained image-to-video (I2V) model and then guide the sampling process using the edited first frame. Generally, a popular choice for maintaining motion and layout from the source video is intervening in the denoising process by injecting attention during reconstruction. However, such injection often leads to unsatisfactory results, where excessive injection leads to conflicting semantics with the source video while insufficient injection brings limited source representation. Recognizing this, we propose an Editing-awaRE (REE) injection method to modulate the injection intensity of each token. Specifically, we first compute the pixel difference between the source and edited first frame to form a corresponding editing mask. Next, we track the editing area throughout the entire video by using optical flow to warp the first-frame mask. Then, editing-aware feature injection intensity for each token is generated accordingly, where injection is not conducted in editing areas. Building upon REE injection, we further propose a zero-shot image-driven video editing framework with recent-emerging rectified-Flow models, dubbed FREE-Edit. Without fine-tuning or training, our FREE-Edit demonstrates effectiveness in various image-driven video editing scenarios, showing its capability to produce higher-quality outputs compared with existing techniques. Project page: https://free-edit.github.io/page/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
