MagicStick：通过控制句柄变换实现可控视频编辑
MagicStick: Controllable Video Editing via Control Handle Transformations
AI总结：
提出MagicStick可控视频编辑方法，利用内部控制信号的关键帧变换实现视频属性编辑，通过扩展模型至时间维度和注意力重混技术，首次展示基于预训练文生图模型的视频属性编辑能力，具有更优的时间一致性和编辑效果。
摘要：
基于文本的视频编辑近期引起了广泛关注，主要用于改变风格或替换具有相似结构的物体。在此基础上，我们证明了形状、大小、位置、运动等属性同样可以在视频中进行编辑。我们的核心洞察是，特定内部特征（例如物体边缘图或人体姿态）的关键帧变换，可以轻松传播到其他帧以提供生成指导。因此，我们提出了MagicStick，这是一种可控视频编辑方法，通过利用对提取的内部控制信号的变换来编辑视频属性。具体而言，为了保持外观，我们将预训练的图像扩散模型和ControlNet扩展至时间维度，并训练低秩适应（LORA）层以适应特定场景。随后，在编辑阶段，我们执行反演与编辑框架。不同的是，在反演和生成过程中均引入了微调后的ControlNet进行注意力指导，并提出了反演与编辑的空间注意力图之间的注意力重混。尽管简洁，我们的方法是首个展示基于预训练文本到图像模型进行视频属性编辑能力的方法。我们在统一框架内展示了大量实验样例。我们还与形状感知的基于文本的编辑和手工制作的运动视频生成进行了比较，证明了我们相比以往工作具有更优的时间一致性和编辑能力。代码和模型可在https://github.com/mayuelala/MagicStick获取。
英文摘要：
Text-based video editing has recently attracted considerable interest in changing the style or replacing the objects with a similar structure. Beyond this, we demonstrate that properties such as shape, size, location, motion, etc., can also be edited in videos. Our key insight is that the keyframe transformations of the specific internal feature (e.g., edge maps of objects or human pose), can easily propagate to other frames to provide generation guidance. We thus propose MagicStick, a controllable video editing method that edits the video properties by utilizing the transformation on the extracted internal control signals. In detail, to keep the appearance, we inflate both the pretrained image diffusion model and ControlNet to the temporal dimension and train low-rank adaptions (LORA) layers to fit the specific scenes. Then, in editing, we perform an inversion and editing framework. Differently, finetuned ControlNet is introduced in both inversion and generation for attention guidance with the proposed attention remix between the spatial attention maps of inversion and editing. Yet succinct, our method is the first method to show the ability of video property editing from the pre-trained text-to-image model. We present experiments on numerous examples within our unified framework. We also compare with shape-aware text-based editing and handcrafted motion video generation, demonstrating our superior temporal consistency and editing capability than previous works. The code and models are available on https://github.com/mayuelala/MagicStick.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
