arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.01427cs.CV
收藏

VideoAnydoor:具有精确运动控制的高保真视频对象插入

VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion Control

The University of Hong Kong(香港大学);DAMO Academy, Alibaba Group(阿里云达摩院);Hupan Lab(虎斑实验室);HUST(华中科技大学)

Yuanpeng Tu, Hao Luo, Xi Chen, Sihui Ji, Xiang Bai, Hengshuang Zhao

AI总结:

本文提出VideoAnydoor框架,通过ID提取器、边界框序列和像素扭曲器,结合文本到视频模型,实现了高保真细节保留与精确运动控制的零样本视频对象插入,无需微调即可支持多种下游应用。

摘要:

尽管视频生成取得了显著进展,将给定对象插入视频中仍是一项具有挑战性的任务。其难点在于既要保留参考对象的外观细节,又要同时准确建模连贯的运动。在本文中,我们提出了VideoAnydoor,一个具有高保真细节保留和精确运动控制的零样本视频对象插入框架。从一个文本到视频模型出发,我们利用一个ID提取器来注入全局身份,并利用一个边界框序列来控制整体运动。为了保留详细外观并同时支持细粒度的运动控制,我们设计了一个像素扭曲器。它以带有任意关键点的参考图像和相应的关键点轨迹作为输入。它根据轨迹扭曲像素细节,并将扭曲后的特征与扩散U-Net融合,从而提高细节保留并支持用户操纵运动轨迹。此外,我们提出了一种结合视频和静态图像并采用加权损失的训练策略,以增强插入质量。VideoAnydoor展示出对现有方法的显著优越性,并在无需特定任务微调的情况下自然支持各种下游应用(例如,说话头生成、视频虚拟试穿、多区域编辑)。

英文摘要:

Despite significant advancements in video generation, inserting a given object into videos remains a challenging task. The difficulty lies in preserving the appearance details of the reference object and accurately modeling coherent motions at the same time. In this paper, we propose VideoAnydoor, a zero-shot video object insertion framework with high-fidelity detail preservation and precise motion control. Starting from a text-to-video model, we utilize an ID extractor to inject the global identity and leverage a box sequence to control the overall motion. To preserve the detailed appearance and meanwhile support fine-grained motion control, we design a pixel warper. It takes the reference image with arbitrary key-points and the corresponding key-point trajectories as inputs. It warps the pixel details according to the trajectories and fuses the warped features with the diffusion U-Net, thus improving detail preservation and supporting users in manipulating the motion trajectories. In addition, we propose a training strategy involving both videos and static images with a weighted loss to enhance insertion quality. VideoAnydoor demonstrates significant superiority over existing methods and naturally supports various downstream applications (e.g., talking head generation, video virtual try-on, multi-region editing) without task-specific fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑