arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.15170cs.LGcs.AI
收藏

扩散采样中Skip-Tuning的惊人有效性

The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling

The Hong Kong University of Science and Technology(香港科技大学);University of Chinese Academy of Sciences(中国科学院大学);Academy of Mathematics and Systems Science(数学与系统科学研究院);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);University of Electronic Science and Technology of China(电子科技大学);National University of Singapore(新加坡国立大学)

Jiajun Ma, Shuchen Xue, Tianyang Hu, Wenjia Wang, Zhaoqiang Liu, Zhenguo Li, Zhi-Ming Ma, Kenji Kawaguchi

AI总结:

本文提出Skip-Tuning,一种针对UNet跳跃连接的无训练调优方法,突破扩散模型ODE采样器在少步采样下的性能限制,仅用19次NFE即在ImageNet 64上使EDM的FID改善100%,并揭示了特征空间损失降低是其有效性的关键机制。

摘要:

随着UNet架构的引入,扩散概率模型已成为图像生成任务中的主导力量。UNet中的一个关键设计是编码器与解码器模块之间的跳跃连接。尽管跳跃连接已被证明能提高训练稳定性和模型性能,但我们揭示出这类捷径可能成为变换复杂度的限制因素。随着采样步数减少,生成过程以及UNet的作用越来越接近从高斯分布到目标分布的推前变换,这对网络的复杂度提出了挑战。为应对这一挑战,我们提出了Skip-Tuning,一种简单却惊人有效的、针对跳跃连接的无训练调优方法。我们的方法在ImageNet 64上仅用19次网络函数评估(NFE)即可使预训练EDM的FID改善100%(达到1.75),突破了ODE采样器无论采样步数多少的限制。令人惊讶的是,当我们增加采样步数时,这种改善仍然持续,并且仅用39次NFE(1.57)就能超越EDM-2的最佳结果(1.58)。我们进行了全面的探索性实验,以揭示这一惊人有效性背后的原因。我们观察到,虽然Skip-Tuning增加了像素空间中的得分匹配损失,但特征空间中的损失却有所降低,尤其是在中间噪声水平上,而这恰好与对图像质量改善最有效的范围相吻合。

英文摘要:

With the incorporation of the UNet architecture, diffusion probabilistic models have become a dominant force in image generation tasks. One key design in UNet is the skip connections between the encoder and decoder blocks. Although skip connections have been shown to improve training stability and model performance, we reveal that such shortcuts can be a limiting factor for the complexity of the transformation. As the sampling steps decrease, the generation process and the role of the UNet get closer to the push-forward transformations from Gaussian distribution to the target, posing a challenge for the network's complexity. To address this challenge, we propose Skip-Tuning, a simple yet surprisingly effective training-free tuning method on the skip connections. Our method can achieve 100% FID improvement for pretrained EDM on ImageNet 64 with only 19 NFEs (1.75), breaking the limit of ODE samplers regardless of sampling steps. Surprisingly, the improvement persists when we increase the number of sampling steps and can even surpass the best result from EDM-2 (1.58) with only 39 NFEs (1.57). Comprehensive exploratory experiments are conducted to shed light on the surprising effectiveness. We observe that while Skip-Tuning increases the score-matching losses in the pixel space, the losses in the feature space are reduced, particularly at intermediate noise levels, which coincide with the most effective range accounting for image quality improvement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑