arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07792cs.CV
收藏

用于高分辨率视频生成的分层 Patch 扩散模型

Hierarchical Patch Diffusion Models for High-Resolution Video Generation

Snap Inc.(Snap公司);KAUST(阿卜杜拉国王科技大学);University of Trento(特伦托大学)

Ivan Skorokhodov, Willi Menapace, Aliaksandr Siarohin, Sergey Tulyakov

AI总结:

本文提出分层 Patch 扩散模型,通过深度上下文融合与自适应计算实现高分辨率视频的端到端训练,并在 UCF-101 上取得最优结果。

摘要:

扩散模型在图像和视频合成中已展现出卓越性能。然而,将其扩展到高分辨率输入颇具挑战,需要把扩散流程重构为多个独立组件,这限制了可扩展性并使下游应用复杂化。这使其在训练期间非常高效,并解锁了高分辨率视频的端到端优化。我们以两种有原则的方式改进 PDM。首先,为增强 patch 之间的一致性,我们开发了深度上下文融合——一种以分层方式将上下文信息从低尺度 patch 传播到高尺度 patch 的架构技术。其次,为加速训练和推理,我们提出自适应计算,将更多网络容量和计算分配给粗略图像细节。所得模型在 UCF-101 256² 的类别条件视频生成中创下 66.32 的全新最优 FVD 分数和 87.68 的 Inception Score,以超过 100% 的幅度超越近期方法。随后,我们展示它可从基础的 36×64 低分辨率生成器快速微调,用于高分辨率 64×288×512 文本到视频合成。据我们所知,我们的模型是首个在此类高分辨率下完全端到端训练的基于扩散的架构。项目网页:https://snap-research.github.io/hpdm。

英文摘要:

Diffusion models have demonstrated remarkable performance in image and video synthesis. However, scaling them to high-resolution inputs is challenging and requires restructuring the diffusion pipeline into multiple independent components, limiting scalability and complicating downstream applications. This makes it very efficient during training and unlocks end-to-end optimization on high-resolution videos. We improve PDMs in two principled ways. First, to enforce consistency between patches, we develop deep context fusion -- an architectural technique that propagates the context information from low-scale to high-scale patches in a hierarchical manner. Second, to accelerate training and inference, we propose adaptive computation, which allocates more network capacity and computation towards coarse image details. The resulting model sets a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in class-conditional video generation on UCF-101 $256^2$, surpassing recent methods by more than 100%. Then, we show that it can be rapidly fine-tuned from a base $36\times 64$ low-resolution generator for high-resolution $64 \times 288 \times 512$ text-to-video synthesis. To the best of our knowledge, our model is the first diffusion-based architecture which is trained on such high resolutions entirely end-to-end. Project webpage: https://snap-research.github.io/hpdm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑