arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.14167cs.CVcs.LG
收藏

T-Stitch:利用轨迹拼接加速预训练扩散模型的采样

T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching

Monash University(莫纳什大学);NVIDIA(英伟达);University of Wisconsin, Madison(威斯康星大学麦迪逊分校);Caltech(加州理工学院)

Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, Anima Anandkumar

AI总结:

本文提出 T-Stitch 轨迹拼接技术,通过在采样早期用小型扩散模型替代大型模型,在不降低生成质量的前提下显著加速预训练扩散模型采样,并适用于多种架构与稳定扩散模型。

摘要:

从扩散概率模型(DPMs)中采样通常代价高昂,难以实现高质量图像生成,并且通常需要大模型执行许多步骤。在本文中,我们提出了采样轨迹拼接技术 T-Stitch,这是一种简单而高效的技术,可在几乎不降低生成质量或完全不降低生成质量的情况下提高采样效率。T-Stitch 并非在整个采样轨迹中仅使用大型 DPM,而是首先在初始步骤中利用较小的 DPM 作为大型 DPM 的低成本直接替代,并在后续阶段切换到大型 DPM。我们的核心见解是,不同的扩散模型在相同的训练数据分布下会学习到相似的编码,并且较小的模型能够在早期步骤中生成良好的全局结构。大量实验表明,T-Stitch 无需训练,普遍适用于不同的架构,并与大多数现有的快速采样技术互补,能够灵活地权衡速度与质量。例如,在 DiT-XL 上,类条件 ImageNet 生成中 40% 的早期时间步可以安全地替换为快 10 倍的 DiT-S,而不会导致性能下降。我们进一步表明,我们的方法还可以作为一种直接替代技术,不仅能够加速流行的预训练稳定扩散(SD)模型,还能改善来自公共模型库的风格化 SD 模型的提示对齐效果。代码已发布在 https://github.com/NVlabs/T-Stitch。

英文摘要:

Sampling from diffusion probabilistic models (DPMs) is often expensive for high-quality image generation and typically requires many steps with a large model. In this paper, we introduce sampling Trajectory Stitching T-Stitch, a simple yet efficient technique to improve the sampling efficiency with little or no generation degradation. Instead of solely using a large DPM for the entire sampling trajectory, T-Stitch first leverages a smaller DPM in the initial steps as a cheap drop-in replacement of the larger DPM and switches to the larger DPM at a later stage. Our key insight is that different diffusion models learn similar encodings under the same training data distribution and smaller models are capable of generating good global structures in the early steps. Extensive experiments demonstrate that T-Stitch is training-free, generally applicable for different architectures, and complements most existing fast sampling techniques with flexible speed and quality trade-offs. On DiT-XL, for example, 40% of the early timesteps can be safely replaced with a 10x faster DiT-S without performance drop on class-conditional ImageNet generation. We further show that our method can also be used as a drop-in technique to not only accelerate the popular pretrained stable diffusion (SD) models but also improve the prompt alignment of stylized SD models from the public model zoo. Code is released at https://github.com/NVlabs/T-Stitch

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑