DreamVideo：用定制主体与运动合成你的梦想视频
DreamVideo: Composing Your Dream Videos with Customized Subject and Motion
AI总结：
本文提出 DreamVideo，利用预训练视频扩散模型，通过身份适配器学习主体外观、运动适配器学习目标运动，实现少量图像与视频条件下的个性化视频生成。
摘要：
使用扩散模型进行定制化生成已在图像生成领域取得令人瞩目的进展，但在具有挑战性的视频生成任务中仍不能令人满意，因为该任务需要同时控制主体和运动。为此，我们提出 DreamVideo，这是一种根据所需主体的少量静态图像以及目标运动的少量视频来生成个性化视频的新颖方法。DreamVideo 借助预训练视频扩散模型，将该任务解耦为主体学习和运动学习两个阶段。主体学习旨在从所提供的图像中准确捕捉主体的精细外观，这通过结合文本反转和对我们精心设计的身份适配器进行微调来实现。在运动学习中，我们构建了一个运动适配器，并在给定视频上对其进行微调，以有效建模目标运动模式。将这两个轻量且高效的适配器相结合，即可灵活地用任意运动定制任意主体。大量实验结果表明，我们的 DreamVideo 在定制化视频生成方面优于最先进方法。我们的项目页面位于 https://dreamvideo-t2v.github.io。
英文摘要：
Customized generation using diffusion models has made impressive progress in image generation, but remains unsatisfactory in the challenging video generation task, as it requires the controllability of both subjects and motions. To that end, we present DreamVideo, a novel approach to generating personalized videos from a few static images of the desired subject and a few videos of target motion. DreamVideo decouples this task into two stages, subject learning and motion learning, by leveraging a pre-trained video diffusion model. The subject learning aims to accurately capture the fine appearance of the subject from provided images, which is achieved by combining textual inversion and fine-tuning of our carefully designed identity adapter. In motion learning, we architect a motion adapter and fine-tune it on the given videos to effectively model the target motion pattern. Combining these two lightweight and efficient adapters allows for flexible customization of any subject with any motion. Extensive experimental results demonstrate the superior performance of our DreamVideo over the state-of-the-art methods for customized video generation. Our project page is at https://dreamvideo-t2v.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
