DreamVideo：具备图像保留能力与文本引导的高保真图像到视频生成
DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidance
AI总结：
针对现有图像到视频生成方法保真度低、时间一致性差的问题，提出基于预训练视频扩散模型、带帧保留分支的DreamVideo方法，结合双条件无分类器引导实现高保真可控视频生成，性能优于当前最优方法。
摘要：
图像到视频生成旨在从给定参考图像出发生成视频，目前已受到广泛关注。现有方法尝试将预训练的文本引导图像扩散模型扩展为图像引导视频生成模型，但这类方法受限于浅层图像引导与较差的时间一致性，往往会出现保真度低或随时间推移出现画面闪烁的问题。为解决这些问题，我们提出了一种高保真图像到视频生成方法DreamVideo，该方法基于预训练视频扩散模型设计了帧保留分支。我们的DreamVideo并非在语义层面将参考图像融入扩散过程，而是通过卷积层感知参考图像，并将对应特征与带噪声潜变量拼接作为模型输入，通过这种方式最大程度保留参考图像的细节。此外，通过引入双条件无分类器引导，只需提供不同的提示文本，就能将单张图像引导生成包含不同动作的视频，这对可控视频生成具有重要意义，拥有广阔的应用前景。我们在公开数据集上开展了全面实验，定量与定性结果均表明我们的方法优于当前最优方法。尤其在保真度方面，我们的模型具备强大的图像保留能力，据我们所知，在UCF101数据集上，与其他图像到视频模型相比，我们的模型取得了最佳结果。同时，通过提供不同的文本提示还能实现精准控制。该模型的更多细节与全面结果将在https://anonymous0769.github.io/DreamVideo/ 中展示。
英文摘要：
Image-to-video generation, which aims to generate a video starting from a given reference image, has drawn great attention. Existing methods try to extend pre-trained text-guided image diffusion models to image-guided video generation models. Nevertheless, these methods often result in either low fidelity or flickering over time due to their limitation to shallow image guidance and poor temporal consistency. To tackle these problems, we propose a high-fidelity image-to-video generation method by devising a frame retention branch based on a pre-trained video diffusion model, named DreamVideo. Instead of integrating the reference image into the diffusion process at a semantic level, our DreamVideo perceives the reference image via convolution layers and concatenates the features with the noisy latents as model input. By this means, the details of the reference image can be preserved to the greatest extent. In addition, by incorporating double-condition classifier-free guidance, a single image can be directed to videos of different actions by providing varying prompt texts. This has significant implications for controllable video generation and holds broad application prospects. We conduct comprehensive experiments on the public dataset, and both quantitative and qualitative results indicate that our method outperforms the state-of-the-art method. Especially for fidelity, our model has a powerful image retention ability and delivers the best results in UCF101 compared to other image-to-video models to our best knowledge. Also, precise control can be achieved by giving different text prompts. Further details and comprehensive results of our model will be presented in https://anonymous0769.github.io/DreamVideo/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
