Omni-Video 2：基于MLLM条件扩散模型的统一视频生成与编辑扩展
Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing
AI总结：
本文提出Omni-Video 2，通过连接预训练多模态大语言模型与视频扩散模型，实现视频生成与编辑的统一。核心方法是利用MLLM生成明确的目标描述以指导生成过程，并通过轻量适配器高效注入多模态条件token，提升复杂编辑任务性能。
摘要：
我们介绍了Omni-Video 2，一个可扩展且计算高效的模型，将预训练的多模态大语言模型（MLLMs）与视频扩散模型连接，用于统一的视频生成和编辑。我们的关键思想是利用MLLM的理解和推理能力生成明确的目标描述以解释用户指令。这样，从理解模型中丰富的上下文表示可以直接用于引导生成过程，从而在复杂的组合编辑任务上提升性能。此外，开发了一个轻量级适配器，将多模态条件token注入到预训练的文本到视频扩散模型中，允许在参数高效的方式下最大限度地重用其强大的生成先验。得益于这些设计，我们通过精心编纂的训练数据将Omni-Video 2扩展到14B的视频扩散模型，支持高质量的文本到视频生成以及各种视频编辑任务，如物体移除、添加、背景更改、复杂运动编辑等。我们在FiVE基准测试集上评估了Omni-Video 2的细粒度视频编辑性能，并在VBench基准测试集上评估了文本到视频生成性能。结果表明，该模型在视频编辑任务中能够很好地遵循复杂的组合指令，同时在视频生成任务中也实现了竞争或更优的质量。
英文摘要：
We present Omni-Video 2, a scalable and computationally efficient model that connects pretrained multimodal large-language models (MLLMs) with video diffusion models for unified video generation and editing. Our key idea is to exploit the understanding and reasoning capabilities of MLLMs to produce explicit target captions to interpret user instructions. In this way, the rich contextual representations from the understanding model are directly used to guide the generative process, thereby improving performance on complex and compositional editing. Moreover, a lightweight adapter is developed to inject multimodal conditional tokens into pretrained text-to-video diffusion models, allowing maximum reuse of their powerful generative priors in a parameter-efficient manner. Benefiting from these designs, we scale up Omni-Video 2 to a 14B video diffusion model on meticulously curated training data with quality, supporting high quality text-to-video generation and various video editing tasks such as object removal, addition, background change, complex motion editing, \emph{etc.} We evaluate the performance of Omni-Video 2 on the FiVE benchmark for fine-grained video editing and the VBench benchmark for text-to-video generation. The results demonstrate its superior ability to follow complex compositional instructions in video editing, while also achieving competitive or superior quality in video generation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
