VIMI：通过多模态指令实现视频生成的基础化
VIMI: Grounding Video Generation through Multi-modal Instruction
AI总结：
针对现有文本到视频扩散模型因缺乏大规模多模态提示视频数据导致视觉基础不足的问题，研究构建多模态提示数据集，提出两阶段训练的VIMI框架，实现多模态理解下的高质量视频生成，在UCF101基准上达SOTA。
摘要：
现有的文本到视频扩散模型在预训练阶段仅依赖纯文本编码器。这一局限源于缺乏大规模多模态提示视频数据集，导致模型缺乏视觉基础，限制了其在多模态整合中的通用性与应用场景。为解决该问题，我们构建了一个大规模多模态提示数据集：采用检索方法将上下文示例与给定文本提示配对，随后利用两阶段训练策略在同一模型内实现多样化的视频生成任务。第一阶段，我们提出一个多模态条件视频生成框架，在这些增强后的数据集上进行预训练，构建具备基础化视频生成能力的基础模型。第二阶段，我们在三类视频生成任务上对第一阶段得到的模型进行微调，融入多模态指令。该过程进一步强化了模型处理多样化输入与任务的能力，确保多模态信息的无缝整合。经过这一两阶段训练，VIMI展现出多模态理解能力，能够生成基于给定输入的、语境丰富且个性化的视频，如图1所示。与以往的视觉基础化视频生成方法相比，VIMI能够合成具有大动作幅度的一致性且时间连贯的视频，同时保留语义控制能力。最后，VIMI还在UCF101基准上取得了当前最优的文本到视频生成结果。
英文摘要：
Existing text-to-video diffusion models rely solely on text-only encoders for their pretraining. This limitation stems from the absence of large-scale multimodal prompt video datasets, resulting in a lack of visual grounding and restricting their versatility and application in multimodal integration. To address this, we construct a large-scale multimodal prompt dataset by employing retrieval methods to pair in-context examples with the given text prompts and then utilize a two-stage training strategy to enable diverse video generation tasks within the same model. In the first stage, we propose a multimodal conditional video generation framework for pretraining on these augmented datasets, establishing a foundational model for grounded video generation. Secondly, we finetune the model from the first stage on three video generation tasks, incorporating multi-modal instructions. This process further refines the model's ability to handle diverse inputs and tasks, ensuring seamless integration of multi-modal information. After this two-stage train-ing process, VIMI demonstrates multimodal understanding capabilities, producing contextually rich and personalized videos grounded in the provided inputs, as shown in Figure 1. Compared to previous visual grounded video generation methods, VIMI can synthesize consistent and temporally coherent videos with large motion while retaining the semantic control. Lastly, VIMI also achieves state-of-the-art text-to-video generation results on UCF101 benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
