arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.12404cs.LGcs.CV
收藏

理解与改进无需训练的基于损失的扩散引导

Understanding and Improving Training-free Loss-based Diffusion Guidance

Microsoft Research Asia(微软亚洲研究院);National University of Singapore(新加坡国立大学)

Yifei Shen, Xinyang Jiang, Yezhen Wang, Yifan Yang, Dongqi Han, Dongsheng Li

AI总结:

本文深入分析了无需训练的基于损失的扩散引导,从优化角度提供理论支持并指出其易受对抗性梯度影响及收敛慢的缺陷,同时提出多项改进技术,在图像与运动生成实验中验证了其有效性。

摘要:

为预训练扩散模型添加额外控制已成为日益热门的研究领域,在计算机视觉、强化学习和AI for science中有广泛应用。最近,多项研究提出利用在干净图像上预训练的现成网络来实现无需训练的基于损失的引导。该方法支持通用控制格式的零样本条件生成,似乎在扩散引导中提供了一种“免费的午餐”。在本文中,我们旨在深入理解无需训练的引导,并克服其局限性。我们提供了从优化角度支持无需训练引导的理论分析,并将其与基于分类器(或无分类器)的引导区分开来。为了阐明其缺陷,我们从理论上证明,与分类器引导相比,无需训练的引导更容易受到对抗性梯度的影响,且收敛速度较慢。随后,我们引入了一系列旨在克服这些局限性的技术,并附带了理论依据和实验证据。我们在图像和运动生成方面的实验证实了这些技术的有效性。

英文摘要:

Adding additional control to pretrained diffusion models has become an increasingly popular research area, with extensive applications in computer vision, reinforcement learning, and AI for science. Recently, several studies have proposed training-free loss-based guidance by using off-the-shelf networks pretrained on clean images. This approach enables zero-shot conditional generation for universal control formats, which appears to offer a free lunch in diffusion guidance. In this paper, we aim to develop a deeper understanding of training-free guidance, as well as overcome its limitations. We offer a theoretical analysis that supports training-free guidance from the perspective of optimization, distinguishing it from classifier-based (or classifier-free) guidance. To elucidate their drawbacks, we theoretically demonstrate that training-free guidance is more susceptible to adversarial gradients and exhibits slower convergence rates compared to classifier guidance. We then introduce a collection of techniques designed to overcome the limitations, accompanied by theoretical rationale and empirical evidence. Our experiments in image and motion generation confirm the efficacy of these techniques.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑