arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.05608cs.CVcs.MM
收藏

基于状态空间骨干的可扩展扩散模型

Scalable Diffusion Models with State Space Backbone

Kunlun Inc.(昆仑公司)

Zhengcong Fei, Mingyuan Fan, Changqian Yu, Junshi Huang

AI总结:

本文提出用状态空间骨干替代U-Net的扩散状态空间模型DiS,将时间、条件和图像块统一作为token处理,在ImageNet上以更低计算成本取得与先前扩散模型相当的性能,并展现出良好的可扩展性。

摘要:

本文对一类建立在状态空间架构之上的扩散模型进行了新的探索。我们致力于为图像数据训练扩散模型,其中传统的U-Net骨干被状态空间骨干所取代,并在原始图像块或潜在空间上运行。鉴于其在处理长程依赖方面的显著有效性,扩散状态空间模型(Diffusion State Space Models, DiS)的特点是将包括时间、条件和带噪图像块在内的所有输入都视为token。我们对DiS的评估涵盖了无条件和类别条件图像生成场景,结果表明DiS在与同等规模的基于CNN或基于Transformer的U-Net架构相比时,表现出相当甚至更优的性能。此外,我们分析了DiS的可扩展性,以Gflops量化的前向传播复杂度来衡量。通过增加深度/宽度或增加输入token数量而获得更高Gflops的DiS模型,一致地表现出更低的FID。除了展示出良好的可扩展性特性外,潜在空间中的DiS-H/2模型在256×256和512×512分辨率的类别条件ImageNet基准上达到了与先前扩散模型相当的性能水平,同时显著降低了计算负担。代码和模型可在以下地址获取:https://github.com/feizc/DiS。

英文摘要:

This paper presents a new exploration into a category of diffusion models built upon state space architecture. We endeavor to train diffusion models for image data, wherein the traditional U-Net backbone is supplanted by a state space backbone, functioning on raw patches or latent space. Given its notable efficacy in accommodating long-range dependencies, Diffusion State Space Models (DiS) are distinguished by treating all inputs including time, condition, and noisy image patches as tokens. Our assessment of DiS encompasses both unconditional and class-conditional image generation scenarios, revealing that DiS exhibits comparable, if not superior, performance to CNN-based or Transformer-based U-Net architectures of commensurate size. Furthermore, we analyze the scalability of DiS, gauged by the forward pass complexity quantified in Gflops. DiS models with higher Gflops, achieved through augmentation of depth/width or augmentation of input tokens, consistently demonstrate lower FID. In addition to demonstrating commendable scalability characteristics, DiS-H/2 models in latent space achieve performance levels akin to prior diffusion models on class-conditional ImageNet benchmarks at the resolution of 256$\times$256 and 512$\times$512, while significantly reducing the computational burden. The code and models are available at: https://github.com/feizc/DiS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑