MagDiff: 面向高保真视频生成与编辑的多对齐扩散模型
MagDiff: Multi-Alignment Diffusion for High-Fidelity Video Generation and Editing
AI总结：
针对单一扩散模型难兼顾视频生成与编辑、文本驱动扩散模态对齐能力不足的问题，提出含三重对齐机制的MagDiff统一模型，在四个基准上表现优于现有方法。
摘要：
扩散模型被广泛应用于视频生成或视频编辑任务。由于每个领域都存在其任务特有的问题，仅靠开发单个扩散模型很难同时完成两类任务。完全依赖文本提示的视频扩散模型可被改造以统一这两项任务，但它在文本与图像的异质模态对齐方面能力不足，会引发各类对齐偏差问题。在本研究中，我们首次提出了一种统一的多对齐扩散模型，命名为MagDiff，可同时用于高保真视频生成与编辑两类任务。所提出的MagDiff引入了三种对齐机制，分别是主体驱动对齐、自适应提示对齐和高保真对齐。具体而言，主体驱动对齐用于平衡图像提示与文本提示，作为支撑两类任务的统一基础生成模型；自适应提示对齐通过为图像提示和文本提示分配不同权重，来凸显同构对齐与异构对齐的不同优势；高保真对齐则将主体图像作为额外的模型输入，进一步提升视频生成与编辑的保真度。在四个基准数据集上的实验结果表明，我们的方法在两项任务上的表现均优于现有方法。
英文摘要：
The diffusion model is widely leveraged for either video generation or video editing. As each field has its task-specific problems, it is difficult to merely develop a single diffusion for completing both tasks simultaneously. Video diffusion sorely relying on the text prompt can be adapted to unify the two tasks. However, it lacks a high capability of aligning heterogeneous modalities between text and image, leading to various misalignment problems. In this work, we are the first to propose a unified Multi-alignment Diffusion, dubbed as MagDiff, for both tasks of high-fidelity video generation and editing. The proposed MagDiff introduces three types of alignments, including subject-driven alignment, adaptive prompts alignment, and high-fidelity alignment. Particularly, the subject-driven alignment is put forward to trade off the image and text prompts, serving as a unified foundation generative model for both tasks. The adaptive prompts alignment is introduced to emphasize different strengths of homogeneous and heterogeneous alignments by assigning different values of weights to the image and the text prompts. The high-fidelity alignment is developed to further enhance the fidelity of both video generation and editing by taking the subject image as an additional model input. Experimental results on four benchmarks suggest that our method outperforms the previous method on each task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
