arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.15829cs.CV
收藏

MVOC:一种基于扩散模型的免训练多视频对象合成方法

MVOC: a training-free multiple video object composition method with diffusion models

Sobey Media Intelligence Laboratory(索贝媒体智能实验室);University of Electronic Science and Technology of China(电子科技大学)

Wei Wang, Yaosen Chen, Yuegen Liu, Qi Yuan, Shubin Yang, Yanru Zhang

AI总结:

针对视频对象合成需兼顾交互效果与运动身份一致性的难题,提出免训练的MVOC方法,通过DDIM反演、首帧编辑及带视频对象依赖模块的图生视频,性能优于现有SOTA方法。

摘要:

视频合成是视频编辑的核心任务。尽管基于扩散模型的图像合成已取得巨大成功,但将该成果拓展到视频对象合成任务并非易事——这类任务不仅要呈现相应的交互效果,还要确保合成视频中的对象保持运动与身份一致性,这是合成物理和谐视频的必要条件。为解决这一挑战,我们提出了一种基于扩散模型的多视频对象合成(MVOC)方法。具体而言,我们首先对每个视频对象执行DDIM反演以获取对应的噪声特征。其次,我们通过图像编辑方法对每个对象进行组合与编辑,得到合成视频的首帧。最后,我们借助视频对象依赖模块中带有特征与注意力注入的图像到视频生成模型来合成视频,该模块是一种用于视频生成的免训练条件引导操作,能够协调合成视频中可能非独立的各对象间的特征与注意力图。最终的生成模型不仅约束生成视频中的对象与原始对象的运动和身份保持一致,还引入了对象间的交互效果。大量实验表明,所提方法优于现有最先进方法。项目页面:https://sobeymil.github.io/mvoc.com。

英文摘要:

Video composition is the core task of video editing. Although image composition based on diffusion models has been highly successful, it is not straightforward to extend the achievement to video object composition tasks, which not only exhibit corresponding interaction effects but also ensure that the objects in the composited video maintain motion and identity consistency, which is necessary to composite a physical harmony video. To address this challenge, we propose a Multiple Video Object Composition (MVOC) method based on diffusion models. Specifically, we first perform DDIM inversion on each video object to obtain the corresponding noise features. Secondly, we combine and edit each object by image editing methods to obtain the first frame of the composited video. Finally, we use the image-to-video generation model to composite the video with feature and attention injections in the Video Object Dependence Module, which is a training-free conditional guidance operation for video generation, and enables the coordination of features and attention maps between various objects that can be non-independent in the composited video. The final generative model not only constrains the objects in the generated video to be consistent with the original object motion and identity, but also introduces interaction effects between objects. Extensive experiments have demonstrated that the proposed method outperforms existing state-of-the-art approaches. Project page: https://sobeymil.github.io/mvoc.com.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑