区域约束的上下文生成用于教学视频编辑
Region-Constraint In-Context Generation for Instructional Video Editing
AI总结：
ReCo通过引入区域约束建模，提升基于指令的视频编辑中编辑区域的准确性和去噪效果，提出新的正则化方法和大规模数据集。
摘要：
最近的上下文生成范式在教学图像编辑中展示了强大的能力，具有数据效率和合成质量。然而，将这种上下文学习应用于基于指令的视频编辑并不容易。如果不指定编辑区域，结果可能会遇到编辑区域不准确和去噪过程中编辑区域与非编辑区域之间令牌干扰的问题。为此，我们提出了ReCo，一种新的教学视频编辑范式，首次在上下文生成过程中探索了编辑区域与非编辑区域之间的约束建模。技术上，ReCo在宽度上连接源视频和目标视频以进行联合去噪。为了校准视频扩散学习，ReCo利用两个正则化项，即潜在和注意力正则化，分别对一步反向去噪的潜在和注意力图进行操作。前者增加了源视频和目标视频之间编辑区域的潜在差异，同时减少了非编辑区域的差异，强调了对编辑区域的修改并减轻了外部意外内容的生成。后者抑制了编辑区域中令牌对源视频中对应令牌的注意力，从而在目标视频中减少新对象生成时的干扰。此外，我们提出了一种大规模、高质量的视频编辑数据集，即ReCo-Data，包含500,000对指令-视频对以促进模型训练。在四个主要的基于指令的视频编辑任务上进行的广泛实验展示了我们提议的优越性。
英文摘要：
The In-context generation paradigm recently has demonstrated strong power in instructional image editing with both data efficiency and synthesis quality. Nevertheless, shaping such in-context learning for instruction-based video editing is not trivial. Without specifying editing regions, the results can suffer from the problem of inaccurate editing regions and the token interference between editing and non-editing areas during denoising. To address these, we present ReCo, a new instructional video editing paradigm that novelly delves into constraint modeling between editing and non-editing regions during in-context generation. Technically, ReCo width-wise concatenates source and target video for joint denoising. To calibrate video diffusion learning, ReCo capitalizes on two regularization terms, i.e., latent and attention regularization, conducting on one-step backward denoised latents and attention maps, respectively. The former increases the latent discrepancy of the editing region between source and target videos while reducing that of non-editing areas, emphasizing the modification on editing area and alleviating outside unexpected content generation. The latter suppresses the attention of tokens in the editing region to the tokens in counterpart of the source video, thereby mitigating their interference during novel object generation in target video. Furthermore, we propose a large-scale, high-quality video editing dataset, i.e., ReCo-Data, comprising 500K instruction-video pairs to benefit model training. Extensive experiments conducted on four major instruction-based video editing tasks demonstrate the superiority of our proposal.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
