SketchVideo：基于草图的视频生成与编辑
SketchVideo: Sketch-based Video Generation and Editing
AI总结：
针对文本和图像控制视频生成与编辑的局限性，本文提出基于DiT模型的SketchVideo，通过草图控制块和帧间注意力实现空间与运动控制，并支持细粒度编辑，实验证明其性能优越。
摘要：
基于文本提示或图像条件的视频生成与编辑已经取得了显著进展。然而，仅通过文本精确控制全局布局和几何细节仍面临挑战，且通过图像支持运动控制和局部修改也存在困难。在本文中，我们旨在实现基于草图的视频生成中的空间与运动控制，并支持对真实或合成视频的细粒度编辑。基于DiT视频生成模型，我们提出了一种内存高效的控制结构，包含草图控制块，用于预测被跳过的DiT块的残差特征。草图绘制在一个或两个关键帧上（可在任意时间点），以便于交互。为了将这种时间上稀疏的草图条件传播到所有帧，我们提出了一种帧间注意力机制，用于分析关键帧与每个视频帧之间的关系。对于基于草图的视频编辑，我们设计了一个额外的视频插入模块，以保持新编辑内容与原始视频的空间特征和动态运动之间的一致性。在推理过程中，我们使用潜在融合来精确保留未编辑区域。大量实验表明，我们的SketchVideo在可控视频生成与编辑方面取得了优越性能。
英文摘要：
Video generation and editing conditioned on text prompts or images have undergone significant advancements. However, challenges remain in accurately controlling global layout and geometry details solely by texts, and supporting motion control and local modification through images. In this paper, we aim to achieve sketch-based spatial and motion control for video generation and support fine-grained editing of real or synthetic videos. Based on the DiT video generation model, we propose a memory-efficient control structure with sketch control blocks that predict residual features of skipped DiT blocks. Sketches are drawn on one or two keyframes (at arbitrary time points) for easy interaction. To propagate such temporally sparse sketch conditions across all frames, we propose an inter-frame attention mechanism to analyze the relationship between the keyframes and each video frame. For sketch-based video editing, we design an additional video insertion module that maintains consistency between the newly edited content and the original video's spatial feature and dynamic motion. During inference, we use latent fusion for the accurate preservation of unedited regions. Extensive experiments demonstrate that our SketchVideo achieves superior performance in controllable video generation and editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
