SAVE:通过结构无关视频编辑实现主角多样化
SAVE: Protagonist Diversification with Structure Agnostic Video Editing
AI总结:
本文提出SAVE,利用运动个性化、运动词、伪光流和伪词解耦运动与外观,解决视频编辑中主角形状差异较大时易失败的问题。
摘要:
在文本到图像(T2I)生成模型迅猛发展的推动下,文本到视频(T2V)生成也取得了显著进步。相应地,修改视频中的物体或改变视频风格等任务已成为可能。然而,以往工作通常在简单且形状一致的目标上表现良好,而在身体形状与原始主角差异很大的困难目标上容易崩溃。本文指出现有视频编辑方法中存在的偏差问题:它限制了新主角的选择范围;我们尝试利用传统的图像级个性化方法来解决这一问题。我们采用运动个性化,从单个源视频中分离运动,然后相应地修改主角。为处理图像与视频之间的天然差异,我们提出具有膨胀文本嵌入的运动词,以恰当地表示源视频中的运动。我们还通过引入一种新颖的伪光流来约束运动词,使其关注与运动相关的适当区域;该伪光流可根据预先计算的注意力图高效得到。最后,我们使用额外的伪词将运动与源视频的外观解耦。大量实验证明了本方法的编辑能力,朝着更加多样和广泛的视频编辑迈出了一步。
英文摘要:
Driven by the upsurge progress in text-to-image (T2I) generation models, text-to-video (T2V) generation has experienced a significant advance as well. Accordingly, tasks such as modifying the object or changing the style in a video have been possible. However, previous works usually work well on trivial and consistent shapes, and easily collapse on a difficult target that has a largely different body shape from the original one. In this paper, we spot the bias problem in the existing video editing method that restricts the range of choices for the new protagonist and attempt to address this issue using the conventional image-level personalization method. We adopt motion personalization that isolates the motion from a single source video and then modifies the protagonist accordingly. To deal with the natural discrepancy between image and video, we propose a motion word with an inflated textual embedding to properly represent the motion in a source video. We also regulate the motion word to attend to proper motion-related areas by introducing a novel pseudo optical flow, efficiently computed from the pre-calculated attention maps. Finally, we decouple the motion from the appearance of the source video with an additional pseudo word. Extensive experiments demonstrate the editing capability of our method, taking a step toward more diverse and extensive video editing.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。