arXiv 2311.009902025-04-14cs.CV

VideoDreamer：基于语言-视频基础模型的Disen-Mix微调实现定制化多主体文本到视频生成

VideoDreamer: Customized Multi-Subject Text-to-Video Generation with Disen-Mix Finetuning on Language-Video Foundation Models

Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Beijing Film Academy（北京电影学院）；Beijing University of Technology（北京工业大学）

Hong Chen, Xin Wang, Guanning Zeng, Yipeng Zhang, Yuwei Zhou, Feilin Han, Yaofei Wu, Wenwu Zhu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有工作局限于单主体文本到视频生成的问题，提出VideoDreamer框架，通过Disen-Mix微调、人在回路再微调及解耦运动定制策略，实现多主体定制化视频生成，并构建MultiStudioBench基准验证其性能。

摘要：

定制化文本到视频生成旨在生成由用户给定主体引导的文本驱动视频，该领域日益受到关注。然而，现有工作主要局限于面向单主体的文本到视频生成，更具挑战性的定制化多主体生成问题尚未得到探索。本文填补了这一空白，提出了一种新颖的VideoDreamer框架，能够生成时间一致的文本引导视频，忠实保留给定多个主体的视觉特征。具体而言，VideoDreamer采用带有时间模块的预训练Stable Diffusion作为其基础视频生成器，借助文本到图像模型的能力生成多样化内容。视频生成器进一步针对多主体进行定制，利用所提出的Disen-Mix微调（Disen-Mix Finetuning）和人在回路再微调（Human-in-the-Loop Re-finetuning）策略，以解决多主体生成中的属性绑定问题。此外，我们提出了一种解耦的运动定制策略来微调时间模块，从而能够生成同时包含定制主体和定制运动的视频。为了评估定制化多主体文本到视频生成的性能，我们引入了MultiStudioBench基准。大量实验表明，VideoDreamer具有显著能力，能够针对定制的多个主体生成包含新事件和新背景等新内容的视频。

英文摘要：

Customized text-to-video generation aims to generate text-guided videos with user-given subjects, which has gained increasing attention. However, existing works are primarily limited to single-subject oriented text-to-video generation, leaving the more challenging problem of customized multi-subject generation unexplored. In this paper, we fill this gap and propose a novel VideoDreamer framework, which can generate temporally consistent text-guided videos that faithfully preserve the visual features of the given multiple subjects. Specifically, VideoDreamer adopts the pretrained Stable Diffusion with temporal modules as its base video generator, taking the power of the text-to-image model to generate diversified content. The video generator is further customized for multi-subjects, which leverages the proposed Disen-Mix Finetuning and Human-in-the-Loop Re-finetuning strategy, to tackle the attribute binding problem of multi-subject generation. Additionally, we present a disentangled motion customization strategy to finetune the temporal modules so that we can generate videos with both customized subjects and motions. To evaluate the performance of customized multi-subject text-to-video generation, we introduce the MultiStudioBench benchmark. Extensive experiments demonstrate the remarkable ability of VideoDreamer to generate videos with new content such as new events and backgrounds, tailored to the customized multiple subjects.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。